
Senate 


COMPUTER PROBLEMS IN 
GOVERNMENT 

Mr. RIBICOFF. Mr. President, the 
General Accounting Office, examining 
computer-related crimes in Federal Pro" 
grams, studied 69 individual cases that 
together totaled more than $2 million in 
losses to the Government. 

The GAO inquiry revealed that com- 
\ . puter fraud is a growing problem in both 
the Government and private sector and 
that, in many instances— no one knows 
how- many— -it. is . .almost impossible to 
V' detect. ■- 

The name of the GAO study is “Com- 
• puter-Related Crimes in Federal Pro- 
grams.” The study is dated April 29, 
•1976. .V 

GAO obtained its information from the 
investigative flies of the C rimin al Inves- 
tigations Division — CID — Command of 
V the Army; the Navy Investigative Serv- 
ice— NIS— the Office of Special Investi- 
gations— OSI— of the Air Force; the Jus- 
tice Department's Executive Office for 
U.S. Attorneys and the FBI; the Office 
of Investigation of the Agriculture De- 
partment; the Internal Revenue Service 
in Treasury; HEW’s Social Security Ad- 
ministration; the Division of Investiga- 
tion of the Interior Department; and the 
Investigation and Security Services of 
the Veterans Administration. 

In the preponderance of the 69 cases, 
criminal prosecutions resulted. 

. GAO auditors cited these instances of 
computer crimes in Government: 

A Defense Department fuel supply em- 
ployee who had helped automate an ac- 
counting system introduced fraudulent 
payment vouchers into the system. The 
computer could not recognize that the 
transactions were fraudulent and issued 
checks payable to fictitious companies set 
up by the employee and his accomplices. 
These checks were sent directly to banks 
where the conspirators had opened ac- 
counts for the companies. The criminals 
then withdrew the funds from the ac- 
counts. Officials estimated the govern- 
ment paid this employee and his accom- 
plices S100;000 for goods and that were 
never delivered. 

A supervisory clerk responsible for en- 
tering claim transactions to a computer-, 
based social welfare system found she 
could introduce fictitious food stamp 


claims on behalf of accomplices and they 
; would receive the benefits. She processed 
more than $90,000 in claims before she 
was discovered through an anonymous 
tip. 

Ah engineer who was no longer em- 
ployed at a computer installation man- 
aged to continue using the equipment for 
his own purposes. Before he was discov- 
ered, he had used more than $4,000 worth 
of computer time. At another installa- 
tion, a programmer used- a self -initiated 
training program to obtain the use of his 
•agency’s computer system. But instead 
of working on the training exercise, he 
was developing his own computer pro- 
grams which he hoped to sell, GAO 
auditors said. • • .’ * 1 - 

The manager of a computer center V 
processing personal information stole - 
some of this data and sold it to private 
firms. Tire private firms, none of which 
were authorized to have such data, used 
the information to promote their prod- 
ucts. GAO said that although the Gov- \ 
ernment did not lose money in this case 
the privacy of individuals whose data 
records were involved was violated. 

At one large Army installation officers 
estimated that 89 percent of all thefts 
may have been computer related. 

In transmitting their report to the 
Congress, GAO auditors said they were 
precluded from being more specific about 
individual instances of computer fraud 
because first, in many instances infor- v 
mation came from raw investigative^ 
files; second, several of the cases re- 
viewed were still open or were about to- 
be prosecuted at the time GAO com- 
pleted its inquiry; - and third, persons 
who had perpetrated computer frauds 
cooperated with GAO but with the un- 
derstanding that they would not be . 
identified. 

GAO auditors said most of the cases 
they studied did not involve sophisti- 
cated ‘attempts to use computer tech- 
nology for fraudulent purposes. Instead, 
GAO . said, these were uncomplicated 
acts which were made easier because 
management controls over the systems 
involved were inadequate. 

Forty-three of the 69 cases of com- 
puter-related crimes were classified by 
GAO as being “fraudulent record initia- 
tion.” Under this category, GAO included 
cases in which. Federal employees', or 
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ing the c omp uter problem they might 


persons employed by Government con- 
tractors, deliberately falsified informa- 
tion from records and documents to be 
fed into computers. Also included in this 
category was the act of falsifying claims 
by reuse of supporting documents pre- 
viously processed.- 

The second category of computer- 
related, crimes is termed “unauthorized 
or inappropriate use of facilities and 
supplies.” This category includes devel- 
oping salable programs on Government 
computers, doing commercial work for 
outsiders on Government computers and 
duplicating files and selling them. 

“Processing alteration or destruction” 
is the third category of computer-related 
crimes studied by GAO. This offense in- 
cludes such crimes as sabotage or alter- 
ing information in the files affecting pay, 
promotion or assignment, and bypassing 
existing controls to enter unauthorized 
changes. 

The final category examined by GAO 
is “misappropriation of output.” In- 
cluded under this section is the misap- 
propriation of returned checks. 

-In connection with its review of com- 
puter-related crime in the Government, 
GAO commissioned the Stanford Re- 
search Institute — SRI — of Menlo Park, 
Calif., to study similar crirpes in the pri- 
vate sector. I 

GAO said the SRI report indicates the 
same types of crimes, occur in both the 
public and private sectors. GAO said in 
both the public and private areas the 
majority of crimes were committed by 
systems users — that is, persons working 
with the computers being abused— but 
the proportion of user crimes is larger 
in Government. 

GAO auditors said the size of the aver- 
age loss in private sector crimes is higher 
than in the Government cases studied. In. 
a review of 144 cases, SRI found the aver- 
age loss in private business to be $450,- 
000. GAO said the average loss in those 
Government cases in which a dollar 
figure was apparent was $44,000. 

GAO said the Government should im- 
prove its management controls over com- 
puters. GAO also pointed out that audi- 
tors of Government computer programs 
should be educated about the prevalence 
and types of computer crimes. GAO said 
that several Government computer audi- 
tors did not know about crimes which 
had been committed in their own pro- 
grams until GAO informed them. 

Another General Accounting Office 
study found that Navy auditors identi- 
fied a computer as being incorrectly pro- 
gramed in 1969 but the computer was 
not fixed for 5 years, during which time 
the machine initiated unnecessary ac- 
tions that cost the Navy $3 million a 
year. 

GAO said one of the reasons the Navy 
gave for the 5-year delay was that Navy 
officials; were concerned that. by correct- 


cause budget reductions. 

Another instance , of computer short- . 
comings, GAO said, could be seen in a 
situation in which Army computers di- 
rected the shipment of radioactive equip- 
ment without requiring the stipulated 
safeguards for proper, handling. 

These examples were cited by GAO to 
demonstrate problems in the Federal" 
Government’s “automated, decisionmak- 
ing computers.” These computers, oper- 
ating without human supervision, an- 
nually initiate payments, purchases and 
other expenditures involving many bil- 
lions of dollars in Government funds and 
resources and people are not required to 
review these actions to determine 
whether they are correct or not. 

In its report, dated April 26, 1976, en- 
titled “Improvements Needed in Manag- 
ing Automated Decisionmaking by Com- 
puters Throughout the Federal Govern- ; 
ment,” GAO concluded; - - ... . ; v • ? ; 

Computers in Federal departments and 
agencies annually issue unreviewed pay- 
ments and other actions involving billions 
of dollars in government assets. These ac- 
tions are often wrong. They can cost the gov- 
ernment huge sums of money; exactly how 
much no one Knows.' ; 

It Is troubling to note the extent to which 
these decisionmaking computers are able to 
decide things on their own. Computer tech- 
nology is progress, of course. But people 
should monitor closely what these machines, 
are up to. For all their heralded memory 
banks and fantastic instant recall, comput- 
ers are still basically beasts of burden. They 
have no inteUigence, except for what infor- 
mation people insert in them. 

“Automated dec’Uonmaklng by computers” 
occurs when computers are programmed to 
make payments, purchase material . and 
otherwise spend money and take actions 
without the'* assistance- of or review by 
•people. • . • 

In their study of automated decision- 
making computers, GAO auditors concluded 
that these kinds .of computers initiate more 
than 1.7 billion payments and other 
actions by government a year without any. 
person evaluating whether they are correct. 
Government automated decisionmaking 
computers issue each year, a minimum of 
unerveiewed authorizations for payment-or 
checks (excluding payroll) totaling $23 bil- 
lion, the GAO report said. 

Unreviewed bills totaling at lease $1& bil- 
lion are . issued annually by automated de- 
cisionmaking computers, the GAO auditors 
said . 

In addition, the GAO said, these same 
computers issue annually unreviewed req- 
uisitions, shipping orders, repair schedules 
and property disposal orders for material 
valued at $8 billion. 

GAO obtained information on 128 auto- 
mated decisionmaking computer programs at 
the Army, Navy, Air Force, Defense Supply 
Agency, General Services Administration, 
Railroad Retirement Board, Veterans Ad- 
ministration and the Departments of Agri- 
culture, Commerce, Housing and Urban De- 
velopment, Interior, Treasury and Health, 
Education and Welfare. 

The GAO auditors cited examples in which 
automated decisionmaking computers had 
resulted in millions of dollars of waste and, 
in one instance, the unauthorized handling 
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of radioactive components for military equip* 

m jn t ~1969> the GAO report said, the Navy’a 
own auditors found that a computer pro- 
gram serving the Navy Aviation Supply 
Office in Philadelphia was inadequately de- 
signed regarding the ability to correctly re- 
flect demand for the purchase and repair 01 

naval aircraft and spare parts. _ . . 

The Aviation Supply Office in Philadelphia 
Is the central manager for all the purchases 
and repair of aircraft and spare parts for 
the entire Navy. The Aviation Supply Office 
is under the Naval Supply Systems Com- 
mand of the Department of the Navy. ■ 

The inadequacy in the automated deci; 
sionmaKing computer program at the Avia- 
tion Supply Office was not corrected. The 
problem was noted in a GAO study Issued 
May 21. 1974 entitled -Better Methods 

Needed For Cancelling Orders For Material 
No Longer Required.” •. 

Again, however, the inadequacy was 
not corrected and the decisionmaking 
computer continued to inaccurately re- 
flect demand for new equipment and for 
repairs on navalaircraf t. Five years Went 
by before the needed correction was 
made. "At least $3 million in annual un- 
necessary costs were initiated by auto- 
mated decisionmaking applications using 
this overstated demand data,” GAO audi- 
tors said. . • .... 

Design of the automated decisionmak- 
ing computers at the Aviation Supply 
Office was developed at the Fleet Mate- 
riel Support Command, Mechamcsburg, 

Pa., which also reports to the Naval Sup- 
ply Systems Command in Washington. 

GAO asked Navy officials why it had 
taken so long to correct the, computer 
inadequacy. The GAO report said : 

The reasons cited by Navy officials for the 
5-year delay, in initiating the modifications 

included: __ • . _ 

Disagreements within the Navy on whether 
all canceled requisitions should result in ret 
ducing record demands. 

High-priority workload at the design ac- 
tivity mandated by higher headquarters lev- 
els in both the Navy and the Department of 

Defense, and „ 

Lack of pressure placed on. the Navy com- 
mand and design activity by tbe inventory 
control points since reduced demands could 
result in budget reductions « [Emphasis 
added.] . 

The Veterans* Administration— V A — 
uses automated decisionmaking comput- 
ers to make monthly payments to more 
th an 185,000 veterans in apprenticeship 
and other on-the-job training programs. 

The VA computers are supposed to be 
programed to make payments at a rate 
that decreases every 6 months, under the 
assumption that an individual veteran’s 
pay from his employer will increase as 
he learns his trade. 

Ann u ally, the VA computers process 
about 1.4 million unreviewed checks for 
more than $225 million in apprenticeship 
and other on-the-job training benefits. 

However, the data submitted to the com- 
puters was incomplete and, GAO audi- 
tors said, cheeks went out at the highest 
levels to the veterans and no progres- 
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sively declining payment system was im- 
plemented. The result, GAO said, was 
potential overpayments of $700,000. 

' Code 8 is the designation the Army 
gives to equipment and spare parts which 
have radioactive components and which, 
therefore, are required to be handled by 
authorized personnel in a stipulated 
manner. * • ' 

GAO said it obtained from the Army 
Audit Agency data concerning the Army 
Electronics Command, Fort Monmouth, 
N.J., which processes each year at least 
250,000 requisitions for material valued 
at a minim um of $250 ‘million. About 35 
percent of the requisitions are reviewed 
by people, GAO said, and the remaining 
65 percent are processed by automated 
decisionmaking computers without re- 
view by people. 

The Army Audit Agency examined 86 
radioactive commodities handled by this 
Command’s automated decisionmaking 
computers and found that 18 of the com- 
modities were processed not with the 
radioactive designation of code 8— but 
instead carried a code 0 rating. Code 0 
means that no special controls or han- 
dling are required, GAO said. 

In addition, the GAO auditors said, 
another 11 radioactive commodities were 
categorized as code 1, the code that In- 
dicates that the item, is scarce, costly or 
highly technical— but not that it is radio- 
active. . 

GAO said the Army Audit Agency also 
studied the application of automated 
decisionmaking computer .technology, at 
five Army inventory control points. The 
Army auditors found the computers were 
often in error jin. deciding where material 
should be. shipped. The result, the Army 
auditors showed, was an annual loss ?f 
$900,000 in unnecessary transportation 
costs. In addition, a total of $1.3 million 
was incurred by the Army in the early 
1970’s due to unnecessary inventory in- 
' creases caused by errors in these same 
computers. . . > . '• • 

The GAO report said that a major 
cause of inaccurate computer tabulations 
in the Government is the massive 
amounts of information fed into the ma- 
chines which lead “input preparers” — 
that is, computer personnel — to make 
mistakes. - v* : 

GAO noted, for example, that the Navy 
Aviation Supply Office in Philadelphia 
receives about 10 million “transaction 
reports” each year, all of which are then 
fed into computers. Transaction reports 
are mainly prepared by Navy facilities 
that receive, store and issue aeronautical 
equipment. 

In addition, GAO auditors estimated 
that during a 12-month period the VA 
Center in Philadelphia prepared more 
than 4 million documents for insertion 
into computers. 
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To ins ur e more accurate automatic 
computer calculations, GAO proposed 
that the Government require selective or 
cyclical monitoring of actions directed by 
automated decisionmaking computers. 
The GAO also recommended that outside 
auditors or independent design teams 
from elsewhere in a given agency be 
called in. to study the design of a com- 
puter program before it is allowed to 
begin making automated decisions. 

A third General Accounting Office 
study found that the Federal Govern- 
ment’s 9,000 computers which are in- 
volved in billions of dollars in transac- 
tions and contain vast amounts of in- 
formation are inadequately protected 
against terrorism, vandalism, program 
alteration, and natural disasters. 

We can see the potential harm in Gov- 
ernment’s failure to adequately protect 
computer facilities when we consider 
what enormous personal tragedies would 
result from serious damage to the social 
security computerized system. Social se- 
curity could not function without its 
computers. It is impossible to. estimate 
the effects on millions of our elderly citi- 
zens whose livelihood depends on social 
security should the computers be de- 

But the potential threat is not limited 
to social security. In terms of Federal 
revenues, for instance, imagine the havoc 
that would result from the destruction of 
Federal tax records. ' 

In addition, the number of veterans 
in this country is larger than ever be- 
fore. Each of these men and women who 
served in the Armed Forces may be re- 
ceiving, or may be entitled to receive, 
benefits from their military service. val- 
liable data and records pertaining to 
their military service — and the benefits 
that accrue from that service — are on 
computer tapes and, in the event of ca- 
tastrophe, could be lost forever. 

Since 1965, responsibility for control of 
computer applications in the Federal 
Government has been shared by the Gen- 
era! Services Administration, the Office 
of Management and Budget, and the De- 
partment of Commerce. 

The GAO report is named * Managers 
Need To Provide Better Protection for 
Federal Automatic Data Processing Cen- 
ters ” It is dated May 10, 1976. 

The GAO report said the total value 
of Government’s 9,000 computers “is 
many billions” of dollars. ■ 

GAO said the value of some of the 
data which is processed on these com- 
puters such as social security records is 
immeasureable. 

GAO auditors said: 

Consequently— protecting equipment and 
• • data. from unauthorized or inadvertent acta 

of destruction, alteration or misuse is a mat- 
ter of inestimable Importance, 


GAO said, for example, that the Na- 
tional Aeronautics and Space Admin- 
istration could not carry out space pro- 
grams without computer applications; 
nor could the Federal Aviation Admin- 
istration control aircraft effectively. 

Computers are used to manage the 
more than half -billion transactions proc- 
essed by the Social Security Administra- 
tion and the 4 billion facts relating to 
the national population compiled and 
managed by the Bureau of the Census, 
GAO auditors said, adding that many 
other Federal agencies rely heavily on 
computer technology. 

Catastrophic losses to Government- 
sponsored data processing Installations 
such as the loss of human life, irreplace- 
able data and equipment have occurred, 
GAO said. In, many of these losses, GAO 
said, additional security measures were 
implemented after the event, 

GAO said information on the physi- 
cal security measures employed at 28 
Federal data processing facilities led its r 
auditors to conclude that Federal data 
processing assets and valuable data ax - e 
not properly protected.: ^ . -• 

GAO recommended : that to provide 
more security over Government auto- 
matic data- processing operations, the 
Office of Management and Budget—- 
OMB— should direct that management 
officials be appointed at Federal instal- 
lations having data processing systems 
and that they be assigned responsibility 
for automatic data processing physical 
security and risk management. 

Reflective of the amount of money 
Federal agencies spend on computers, 
GAO said, is .the fact that more than $19 t 

billion is expended each year to buy and 

operate Federal data processing systems. 

In concluding that security safe- 
guards are inadequate regarding com- 
puters, GAO studied security techniques, 
at 28 data processing installations of the 
Departments of the Army, Navy, Air 
Force, Agriculture, Transportation, 
State and Health, Education and Wel- 
fare and the Veterans’ Administration. 

Besides the 28 Federal data process- 
ing sites, GAO auditors also studied se- 
curity problems identified at 23 addi- 
tional Government computer installs- 
tions 

In addition, GAO examined data proc- 
essing security systems used at Govern- 
ment ‘contractor sites; universities, pri- 
vate companies, a bank, and a local gov- 

eminent. „ .. 

GAO said major areas of security cov- 
ered in its investigation of data process- 
ing facilities included steps taken by 
management to guard against threats of 
modification or destruction to the physi- 
cal Plant, personnel, computer- hardware 
and software, and to the data being proc- 
essed or stored by the computerized sys- 
tems. " 
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Eighteen of the 28 data processing in- 
stallations were in the . continental 
United States. The remaining 10 were 
abroad. 

Among its findings that compute? in- 
stallations are not properly protected, 
GAO noted that— ' ’ 

Fourteen installations had combustible 
paper supplies or magnetic tape files 
which .were stored in computer rooms 
which exposed systems to losses from 
fire. . • ,. 

Three installations had computers 
which were in use in areas where only 
portable fire extinguishers were avail- 
able. ■ . 

One installation’s computers were in 
operation where no portable fire extin- 
guishers were available. 

Twelve installations had computers 
which were in use above raised flooring 
without periodically cleaning below such 
flooring, constituting a fire hazard. . 

Six installations had computers which 
were in operation where master electri- 
cal power shutdown controls were not 
easily accessible at exit points. 

Teh installations had computers in 
operation in areas where overhead water 
or steam pipes— excluding sprinkler sys- 
tems — existed with inadequate provision 
for drainage. 

Two installations had computers 
which were used in basements below 
ground level, exposing systems to. poten- 
tial flooding conditions. 

Seven installations allowed vendor 
service personnel near computer banks 
without supervision. 

Five installations allowed in-house 
service personnel to move about without 
supervision in computer areas. 

Three installations located computers 
in quarters that were vulnerable to van- 
dals. 

Five installations managed their com- 
puters in ways susceptible to theft or 
misuse. Remotely located computer sys- 
tems were in operation without controls 
to detect improper or erroneous attempts 
to use computers or data files. 

Fourteen installations lacked contin- 
gency pla nnin g. Computerized systems 
were in operation without formal con- 
tingency plans to insure continuity of 
operations if an event occurred that 
threatened security. 

GAO studied instances in which major 
data processing facilities had been hit 
by terrorism, vandalism, fire or natural 
dissstor 

GAO said attempts at sabotage of 
computer activities have been made by 
emnloyees within data processing cen- 
ters. GAO said four attempts had been 
made to sabotage computer operations 
at Wright-Patterson Air Force Base 
near Dayton, Ohio, during a 6-month 
period ending November 15, 1974, by us- 
ing magnets, loosening . wires on . the 
computer mainframe and gouging equip- 
ment with a sharp fool. 


On August 24, 1970, a bomb exploded 
outside the Sterling Hall Building at the 
University of Wisconsin. This building 
housed the Army IVfathematics Research 
Center and other federally funded re- 
search. activities. One employee was 

killed and three others were injured. The - 

explosion damaged 25 buildings at the 
university ' and resulted in a total loss 
of $2.4 million for buildings and equip- 
ment. Computers at the Army Mathe- 
matics Research Center were damaged 
and some programing efforts and 20 
years’ accumulated data was destroyed. 

It has been estimated that this research 
data represented more than 1.3 million 
staff hours of effort. GAO calculated this 
effort to represent an. investment of $16 

million. ~ 

In May of 1972, a bomb exploded on 
the fourth floor of the Pentagon above 
the computer facility and caused extern- ; 
sive damage. The computer facility was 
flooded from, broken water pipes and 
parts of it were inoperable for about 29 
hours. ' 

The computer center at the National 
Institutes of Health, Bethesda, Md., has 
experienced many computer system fail- 
ures due to electrical power failures. 
GAO said officials of the computer center 
estimated that they lost a minimum of 
$500,000 annually from electrical power 
fluctuations. During a 15-week period, 
the NIH computer center experienced 
6 major electrical power fluctuations 
which caused 15 computer system fail- 
ures. These failures resulted in destruc- 
tion of data for 375 batch processing jobs 
and for 2,250 remote terminal users. 
GAO said these power fluctuations 
caused replacement of electronics cost- 
ing more than $94,000 in various compo- 
nents of the computer systems. 

On June 24,- 1972, water from the Sus- 
quehanna River flooded all of downtown 
Wilkes-Barre, Pa., and filled the base- 
ment of the post office building. Water 
continued rising until about 6 inches 
of it were on the computer room floor. 
About $7.5 million worth of .Govern- 
ment computer equipment was located 
on raised flooring on the first floor. Had 
the water risen about an inch more it 
would have ruined virtually all of the 
computer equipment, GAO said. 

GAO described a 1959 fire at the 
Pentagon which, destroyed three com- 
plete computer systems valued at $6.5 
million. The fire, started in a vault con- 
taining stored paper and magnetic tape 
and spread throughout the computer 
center. When the first occurred em- 
ployees were unable to reach the switch 
to turn off electrical power for the com- 
puter system. This created a hazardous 
situation for firefighting efforts. 

GAO cited another. example of cata- 
strophic loss Caused by fire to a Govern- 
ment facility, although computer rec- 
ords were not directly involved. In July 
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of 1973, fire broke out in the Military 
Personnel Records Center in St. Louis, 
Mo. Sections of the b ui ld in g housing 
these records were not equipped with 
sprinkler systems, smoke detectors or 
fire walls. Although the fire did major 
damage to papers and not computerized 
records, GAO said, it nevertheless illus- 
trated how devastating the loss of irre- 
placeable documents and records can be. 
GAO said that since such records are 
being put on computers more and more, 
the problem increasingly becomes a com- 
puter security problem. 

GAO said the St. Louis records center 
has been the repository for about 52 mil-? 
lion records* on military personnel ac- 
tions since 1912. The sixth floor, where 
the fire started, contained about 22 mil- 
lion military personnel files or jackets. 
About 16.8 million of these records were 

lost. ... 

Of the St. Louis fire, GAO auditors 

saidr v 

This installation’s mission is to maintain 
these official government records and to re- 
spond to. Inquiries made by the Congress, 
other government agencies and the taxpayer. 
This mission will now be hampered for some 
time because the lost records-—* some of which 
may be irreplaceable-must be reconstructed 
to satisfy inquiries, which is a costly and 
time-consuming process. 

While it is unreasonable to expect that 
there would be backup for every : original 
record in the manual files, it is reasonable 
to assume that some ' sort of contingency 
planning should have been done to insure 
continuity of operations when a loss has 
occurred. Agency officials told us that a con- 
tingency plan was formulated after the fire 
happened. 0 

■ GAO cited an instance at Kelly Air 
Force Rase in San Antonio, Tex., in wMch 
someone altered a computer program 
that resulted in a $100,000 theft of Gov- 
ernment money. Due to the computer 
alteration, the Air Force paid $100,000 to 
bogus companies for aircraft fuel never 
delivered. The bogus companies were es- 
tablished by a Government employee 
working at the base. The employee had 
indepth knowledge of the computerized 
fuel accounting system which he helped 
develop and install. An investigation was 
begun when a bank contacted the Air 
Force regarding suspicious banking 
transactions involving Government 
checks. The employee was arrested, con- 
victed and sentenced to 10 years in 

P Among the agency comments to the 
GAO report were these: • 

James T. Lynn, Director of the Office 
of Management and Budget, said the 
GAO report was correct in citing a need 
for greater awareness of threats to phys- 
ical security” in automated data process- 
ing. However, Lynn said OMB did not 
support GAO’s recommendation that an 
official in each agency be assigned re- 
sponsibility for computer security,. In- 
stead, Lynn said, the head of each 
agency should tlccids iiow computer sb.ig 


guards should be provided and who I 
should be in charge. I - 

Terence E. McClany, Assistant Secre- 
tary of Defense, Comptroller, said of tne 
GAO report that in general, “the impor- 
tance of the subject, the general siib- 
stance of the report, and.the thrust of the 
recommendations are wholeheartedly 

endorsed * * *” ’ ■ ' 

John D. Young, Assistant Secretary of 
HEW, Comptroller,. said. We full con- 
cur with the recommendations contained 

in the report. . . • . . ■ _ 

William S. Heffelfinger, Assistant Sec- .. 

retary for Administration in the Depart- 
ment of Transportation, endorsed the . 
GAO study. - .. M ■ 

The GAO report did not identify any 
of the specific installations where it dis- , 
covered inadequate safeguards against 
computer damage. GAO auditors felt 
that to identify these sites would be to 
run Ihe risk that persons might wish to 
exploit these security weaknesses. 

Mr. • President, as chairman of the 
Senate Committee on Government Op- 
erations, X have directed the staff to 
conduct a preliminary inquiry into the 
problems associated with computer- 
related crimes in Federal programs, au- / 
tomated decisionmaking computers in . 
Federal programs and computer security 
in Federal programs. 

Also in connection with computer : 
problems in the Federal Government, 
Rebecca Leet in the Washington Star of 
May 10, 1976 has written an informative 
article. Printed on page 1 of the Star, 
the headline of the article is “Two GAO 
Studies Criticize Lack of Controls on 
Computers.” Mr. President, I ask unan- 
imous consent that the Washington Star 
article by Ms. Leet be printed in the 
Record. 

There being no objection, the article 
was ordered to be printed in the Record, 
as follows: > ■ - > 

< c Two GAO Studies CsracusB Lack 

OS' CONTEOLS OSf COMP-errHFtS . ■ ■ - ’ 

(By Kebecca Leet) * 

The rapid movement of the federal govern- 
ment to greater and greater reliance on com- 
puters baa not been accompanied by controls 
to assure that computer orders are appro- 
priate or necessary, according to two General 
Accounting. Office reports, - 


The result is a government highly sus- 
ceptible to being defrauded, even by un- 
sophisticated workers and to losing millions 
.of dollars annually in overpayments, un- 
necessary repairs and the like. 

Probably every federal agency which uses 
computers lacks the controls necessary to 
prevent the kind of computer fraud and mis- 
takes which led to the $622 million overpay- 
ment in federal welfare benefits the Social 
Security Administration has made since 1974. 

The Washington Star reported on Friday 
that audits by the Department .of Health, 
Education, and Welfare had found that lax 
management of the Social Security com- 
puter system left large amounts of money 
exposed to errors and fraud. 
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The GAO in 

agency after nufiTerops re^-- 
fraud iri private industry, are the first time 
the federal government has looked at what 
steps it has taken to protect Itself against 
computer fraud and mistakes since the gov- 
ernment began using them in 1952. Cur- 
rently, 9,000 computers are used by the fed- 
eral government. 

What GAO found. In most cases, is that 
government agencies have been more con- 
cerned with getting a computer program un- 
der way by the date promised than they have 
in seeing It function properly, according to 
the reports. 

Ken Pollock, a GAO official who deals with 
computer policy, said the agency was “ap- 
palled” by the lack of controls It found. "In 
the old manual systems, everyone was very 
conscious of controls. * • . In the hurry to get 
automated, these things were ignored," he 

said. _ . 

The report on computer fraud noted that 
the control weaknesses which criminals were 
taking advantage of "are mostly basic man- 
agement controls long recognized as being 
necessary to insure proper operations.” 

The rush to get the Social Security Ad- 
ministration’s Supplemental Security In- 
come (SSI) program instituted by its target 
date of Jan. 1, 1974, has been given as the 
.main reason why the program has so many 
hugs in it. The Star previously disclosed that 
$622 has ’ been overpaid the country’s aged, 
blind and disabled welfare recepients under 
SSI. 

Once a government computer system is in 
operation. Pollock noted, "there’s always 
something else (for programmers) to do 
other than go back” and review the system 
to see if it is functioning properly. 

pollock said that GAO had difficulty in 
making its two studies because fraud by com- 
puters had never before been isolated from 
regular fraud and because no one had ever 
isolated the process which GAO called "auto- 
mated decision-making by computers.” 

Automated decisionmaking by computers 
occurs when a computer is programmed to 
issue checks or hills or orders for an agency 
under certain circumstances and the actions 
are taken without humans ever reviewing 
them to see if they are correct. 

GAO discovered that such computer pro- 
grams annually issue payments or checks— 
excluding payrolls— totaling $26 billion. They 
issue bills totaling $10 billion and issue re- 
quisitions, shipping orders, repairs and dis- 
posal orders for materials valued at $8 billion. 
Humans never review any of those actions. 

GAO said that while it believes most of 
automatic decisions the computers are cor- 
rect, "we know from audit reports we re- 
viewed that they also make bad decisions 
that cost the government many millions of 
dollars annually. Additionally, '• bad deci- 
sions . . . may result m harm to people.” 

"There la no federal-wide policy, guidance 
or other instructions on how* computers is- 
suing unreviewed actions should be managed 
by federal agencies,” the report said. "There 
is little checking or monitoring of output on 
an ongoing or short-term periodic basis. 

"Internal audit reviews of these computer 
actions are- made, sporadically or not at all,” 
the report said. 

Examples of the mistakes 'of such auto- 
mated decision-making ■ by computers, as 
noted in the GAO report, include: 

The Navy yearly spent $3 million to per- 
form unnecessary airplane overhauls from 
1969 to 1974 because incorrect information 


Ire ‘ 

(As the GAO report notes, once computer 
mistakes are discovered, they must be cor- 
rected to change the faulty outcome. The 
Navy resisted correcting this* program error 
for two years after It was discovered at least 
partly, GAO was told, because it feared its 
budget would be reduced if a lower use level 
was shown.) 

A faulty computer program led to the un- 
needed cross-country shipment of $1 million 
worth of Army supplies one year. 

In a study of 89 such shipments, mistakes 
in a computer code resulted in 29 improper 
shipments of radioactive material. The ma- 
terial was shipped without proper safeguards 
and "there was doubt,” the report noted, that 
customers in some cases should have received 
the material and in others that they knew It 
was radioactive. 

Regarding computer frauds the GAO said 
that contrary to the general assumption, 
those defrauding the government by com- 
puters are mainly the untrained, relatively 
unsophisticated computer users and not the 
highly trained computer programmers. 

In 50 of 69 instancesrof computer fraud 
was committed by the computer user who 
knew which keys to press on a computer 
terminal to get a check Issued, to someone. 
Since no records are kept of computer fraud 
separate from regular fraud, the 69 instances 
which the GAO investigated, Pollock said, 
were all cases which government investiga- 
tors recalled as having involved computers. 

These instances, which occurred since 
1970, resulted in the federal government 
being bilked out of $2 million. It was rela- 
tively easy to accomplish, GAO found, be- . 
cause of inadequate ^controls over the com- - 
puter' systems. ... . * 

Of the 69 cases, investigators reviewed 12 
in depth. Their conclusions went largely un- 
challenged by the agencies which were vic- 
tims of the fraud, according to the report. 

"In every case we reviewed in detail, the 
incidents were directly traceable to weak- 
nesses in the system controls „ „ . The pri- 
mary reason weaknesses in system controls 
existed was that management failed to rec- 
ognize the Importances uf controlling sys- 
tems,” the report said. 

"Managers . . . primarily emphasized 
their systems operational; control was not 
emphasized.” - y - \ - y 

^ "Computer criminals were typically not 
‘professional’ criminals, but persons who 
encountered difficulties on a short-term 
basis and who commit their crimes to help 
them solve their problems. They experience 
great personal suffering when their acts are 
discovered. Therefore, a highly visible and 
active audit function could dissuade them 
from attempting crime,” the report noted. 

The Instances of fraud, included amounts 
ranging from $320 to $250,000. Some of the 
fraud was not in bilking the government of 
money but in using government computers 
to design programs which were then sold to 
commercial firms. 

Conclusions of the GAO reports, which 
were sent to all government agencies using 
computers, are not' binding on agencies. 

However, agencies must Inform Congress I 
within 60 days what actions they have taken 
as a nesult of the reports’ recommendations 
on Implementing controls. It is then up to 
Congress to decide whether legislation is j 
needed to correct the deficiencies noted. 
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CHAPTER I 

• ' iNIRODUcyroN 

Many early business applications- on computers involved 
ent' ring, manipulating, ‘ and summarizing data and generating 
.roo iris. Most output; or oducec by these computers was 
manually reviewed (1) for correctness and/or (2) tc ’decide 
what actions should be taken on the basis of. the out out 
ropurt. • • . 

As jrore complex computer processing developed:,, 'the 
applications became more innovative. Computers were assign- 
ed certain repetitive decisionmaking work which duplicated 
steps people had. taken to do the job previously* -The output 
of these computers is frequently not reviewed by people 
(that isi no manual review). . 


These types of applications have no establishes name. 

We are calling them automated decisionmaking ’ applications* 

AU'r0: : iATED,DEClSI0i!aAKISG_APPLICAT10NS 

Automated decisionmaking applications are compete r 
programs that initiate actioii (though output) on the basis 
of programmable decisionmaking criteria established - ! ?:/ 
management and incorporated in computer instruc tion .. " Th e 
"distinguish r ng characteristic of these applications, as co>'l 
pared .to other computer application programs, is that mapH’* 
of the computer’s actions take place without manual revirwr 
and evaluation. .... 

1 ' ■ . 

An inventory application is an example of a comouter. 
application program. If the computer processing of a requi- 
sition for -material reduces the onhand quantity, below the 
.reorder point and if the computer issues a purchase order 
without anyone reviewing the proposed procurement quantity, 
then the application is an automated decisionmaking appli- 
cation.' Some of the computer output of .these applications 
is reviewed. In the foregoing example, the application 
may call for manual reviews. of quantitifces ert all. purchase 
orders over $5,000, with all purchase orders under that, 
amount being released without review. 

he reviewed these applications because (1) billions 
ot dollars ore involved in the on reviewed actions that 
they initiate and (2) of indications that funds were being 
wasted because of incorrect actions. 





car 
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CHARA C TERISTICS 

One objective of. using computers operating under auto- 
mated decisionmaking applications is tc take advantage of 
the.ii speed , accuracy, storage capabilities, anc capacity 
to obey" predetermined instructions. . • These .appl Scat ions 
are needed in part, because of the tremendous volumes of 
information 'to be obtained, manipulated (processed), analyz- 
ed, and acted on in carrying out agency missions and .joals. 

Automated decisionmaking applications process large 
volumes. of transactions put into the computer system from 
various sources. They make repetitive decisions that, in 
many cases, previously have been made by people.. The. 
decision inductions, built into the program,’ ask questions 
about the transactions and then initiate many actions through 
output. The actions depend soiely on the criteria {logic) 
and data inside the computer system. • 

Computer program ’ . 

The computer program (software), written by people, •' 
instructs the computer (1) to examine- the input data and/ 
or data already in automatic data' processing (ADP) files, 

(2) to perform logical decisionmaking steps and computa- 
tions in processing the data, and (3) to initiate actions 
in the form, of output as a result of this process. 

s' - . ■ Input ' * • • ‘ . 

• Data is usually obtained, by people from var ious sources ' 
and is put ir.tp the computer in; machine-readable form 
(including punched cards," optical character, .recognition 
documents, paper tape, magnetic ink character : recognition 
documents, and direct keyboard entry). The data can be 
entered directly for processing or can. be recorded on ADP 
files for processing at. a later tinr. '• ' 

Output . . . 

. . The application outputs are such -things as (1) direc-~__ 

tives "to act (such as orde rs to ship material), (2) payment 
■ . aTTth o7T~zaT ion s ; or "~cb¥cK~s , *TT) bills, and (~4 )~Ti o 1 1 ce s . A 

large percentage "of the output of these applications is not 
. . manually reviewed and evaluated by people. 

The following illustration shows a. computer operating 
under automated decisionmaking appl ications. 



L 
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The form of output varies (including listings, magnetic 


tapes, preprinted forms, and punched cards). These outputs 
indicate the decisions resulting from computer processing 


directed by the software. . 


7 


' ' The outputs that are not reviewed or evaluated are 
.usually issued to the organizations and people which take 
the action being directed or which are being paid, billed,, 
or not i f ied. • • • . • . . 


• Some of- the output of many automated dec.i si onriak i ng 
applications is manually reviewed. Under -management by 
except i on* principles, some output the nature and extent 
of which is- determined by management, is sent to people i.i 
the organization for manual review and evaluation. This 
technique allows people to consider criteria, factors, and 
information not contained in the computer system in deciding 
whether the computer-directed action should be taken. For 
these applications manual intervention takes place only for 
the actions output for review, . ; . - 


The criteria for directing manual review of the output 
are contained in t her- decisionmaking, part . of the program. In. 
the inventory application example;, the program would direct 
that purchases over $5,000 be output for manual revievx. The 
applications can be programed so that none of. the output will 
be manually reviewed or evaluated before actions ass taken. . 

CONTRAST WITH OTHER COMPUTER APPLICATIONS . 

. Application programs designed to provide output to peo- 
ple for. information and analysis .are not automated decision- 
making applications. Many. types of these application pro- 
grams are used in Government, and the outputs are.' sent to 
people for review before actions are taken. 

Typical application programs that are not automated 
deci sr onmak ing applications include: 

— Systems that make recommendations,, all of. which are 
• manually reviewed before actions are takeni 

--Management .and other information systems, which pro- 
vide data to various revels of managers to. assist 
them in making pol icy, management , .arid operating 
decisions. 


--Most mathematical models. 
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COMPTROLLER GENERAL OF THE UNITED STATES 

WASHINGTON. DC, S9WS 


B-115369 


To the President of the Senate and the 
Speaker of the House of Representatives 

Many Federal . agencies use computers to initiate actions 
are not reviewed by people. Thii 

™f r0bl ?* S that have 566,11 ex P er ie.nceo by agencies that use 
computers this way arid offers some 
solve them. 


that 

many 


lo report, describes the 

the 

suggest ions cm how t o 


Act. 


l| 2 T a Mi 0 r r c 5 r Ud L? UrSUant to the 3ud S-t and Accounting 
, J1 L.S.C. 53), and the A< 


of 1950 (31 U.S.C. 67 t. 


Accounting and Audit ing Act 


nfcio« e ,J r « S6rdlng copies of this report to the Director, 
Office of Management and Budget ; the Secret ary of Commerce 

deL?tnin!^ f tr ^° r a f Ge " eral Services; and the. heads of Fedet 
departments and independent agencies. 


al 



Compt roll er General 
of the United States 
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COMPTROLLER GENERAL’S 
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DIGEST 


IMPROVEMENTS HEEDED IN 
MANAGING AUTOMATED 
DECISIONMAKING BY COMPUTERS 
THROUGHOUT THE FEDERAL 
GOVERNMENT 


Federal agency computers cause more than 1.7 
billion payments and other actions a year with- 
out anybody reviewing or evaluating whether 
they are correct. Many agencies use computers 
in this way. At a minimum. Government computers 
issue annually: 

Unreviewed authorizations for payments or 
checks (excluding payroll) totaling $26 bil- 


Unreviewed bills totaling $10 billion. 

—Unreviewed requisitions, shipping orders, 
repair schedules, and disposal orders for 
material valued at $8 billion. 


COMPUTERS CAN ISSUE 
INCORRECT' ACTIONS 


Computers ore complex data processing machines 
which are indispensable to the day-to-day oper- 
ations of most Federal agencies. They can proc- 
ess data quickly and are especially useful in 
business-type applications which involve repet- 
itive processing of large volumes of data. How- 
ever, computer actions are only as good as the 
computer programs (or software) that make the 
computers operate and the data within the sys- 
tern. Computers can cause incorrect actions if 
these factors are wrong. The result is over- 
payments and unnecessary or premature costs. 


Some agencies’ internal audit reports show that 
unreviewed incorrect actions have been issued 
by several Government computers, incurring over- 
payments and unnecessary or premature costs of 
tens of millions of dollars annually. For ex- 
ample: • 


Computers of one military department incurred 
increased inventory pipeline and transporta- 
tion costs of $2.2 million because of erro- 
neous software. (See p. 13.) 


Xft.sr Sfrto ef. Upon removal. the report 
cover date should be hereon. 
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— One military agency's computer caused mil- 
lions of dollars in unnecessary and/or 
premature overhaul of equipment because of 
software and data problems. (See p. 14.) 

Computers issuing incorrect actions over an 
extended period of time increase the impact of 
overpayments, unnecessary costs, and so on. It 
is important to detect incorrect actions. It 
is equally important to correct them as early 
as possible. 

In this report, software that instructs com- 
puters to issue unreviewed actions are being 
called automated decisionmaking applications. 

CAUSES FOR INCORRECT 
COMPUTER ACTIONS 

Incorrect computer actions occur because of 
software problems and/or data problems. The 
causes of these problems are numerous. 

Software problems, for example, can be caused 
by inadequate communications between people 
involved in software development. (See pp. 20 
to 27. ) 

Data problems, for example, can be caused by the 
use of input forms that are too complex. (See 
pp. 29 to 32. ) 

FEDE RAL POLICY AND 
AGE'NCY MANAGEMENT 

There is no Federal-wide policy, guidance, or 
other instructions on how computers issuing 
unreviewed actions should be managed by Federal 
agencies. There is little checking or monitoring 
of output oh an ongoing or short-term periodic 
basis. Internal audit reviews of these computer 
actions are made sporadically or not at all. 

Several things can be done that will disclose 
some of the problems before they occur and/or 
before computers make decisions that can cause 
incorrect actions for an extended period. 

These practices should be considered for 
Government-wide use. (See pp. 47 to 49.) 


ii 
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RECOMMENDATIONS 

GAO believes that, since automated decision- 
making applications have not. previousl -’ been 
recognized as a separate problem area requir- 
ing management attention and since millions 
of dollars are presently being wasted as the 
result of actions generated by such systems, 
the Office of Management and Budget should 
act. immediately to improve the situation. 
Specifically, GAO recommends that, the Direc- 
tor , Office of Management' and Budget, in his 
oversight capacity, require that: 

— Each agency determine whether any of its 
computer operations involve automated de- 
cisionmaking applications. 

--The agencies review each operation to de- 
termine whethet incorrect actions are being 
taken as a result of these applications. 
(Pending issuance of technical guidelines 
by the National Bureau of Standards for 
making such reviews, the agencies should ex- 
amine enough automatically generated deci- 
sions to provide a basis for deciding 
whether incorrect decisions ate occurring 
and, if so, should take the necessary steps 
to correct the situation causing the in- 
accurate decisions.) 

— -Before any new automated decisionmaking ap- 
plications are initiated by an agency, the 
proper steps are taken to insure correct de- 
cisions. This would include, pending is- 
suance of National Bureau of Standards guide- 
lines, a carefully chosen combination of in- 
dependent review of systems design, adequate 
testing before implementation, and periodic 
testing of decisions after' implementation, as 
discussed in this report. 

— Agencies make reports on the actions taken, 
and establish an appropriate mechanism for 
monitoring reports. 

GAO * .. , ..mse' the 'National Bu- 

re. t- . Standards has responsibilities for 
technical aspects of automatic data proc- 
essing, the Secretary of Commerce direct the 
Bureau to issue technical guidelines for de- 
veloping, using, technically evaluating. 
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documenting, ana modifying these applications 
in the Federal Government'. When issued, these 
guidelines should contain certain criteria for 
independent technical reviews and for monitor- 
ing of these applications to insure problems 
are detected and corrected promptly. The Gen- 
eral Services Administration should incorporate 
the Bureau guidelines in its agency directives. 

In addition, GAO recommends that: 

— As the General Services Administration sug- 
gested, the Civil Service Commission develop 
and add to its automated data processing, 
training curriculum courses in automated de- 
cisionmaking applications so that, managers, 
technical personnel, and auditors will be- 
come better equipped to deal with them in 
an appropriate manner. 

— Internal audit grcups in agencies having au- 
tomated decisionmaking applications partici- 
• pate actively in design, test, and reviews 
of such systems to carry out their respon- 
sibilities. 

Finally, GAO' suggests .that, - the Joint Financial 
Management. Improvement r-t ogram consider this 
area for ongoing attention. 

GAO is sending copies of this report to all 
departments and independent agencies for their 
information, use, and ^siidance pending issu- 
ance of the Office of Management and Budget, 
and the National Bureau of standards material. 

GAO received comments from several agencies. 
They agreed in principle to the need for in- 
creased management attention to automated 
decisionmaking applications. (See pp. 53 to 
55.) 
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CHAPTER /I 

• . INTRODUCTION 1 

Many early business applications- on computers involved 
entering, manipulating, and summarizing; data and generating 
.reports. Most output or oduced by these computers was 
manually reviewed M) for correctness and/or (2) to decide 
what actions should be taken on the basis of; the output 
report. 

As more complex computer processing developed , the 
applications became more innovative. Computers were assign- 
ed certain repetitive decisionmaking work which duplicated 
steps people had taken to do the job previously* The output 
of these computers is frequently not reviewed by people 
(that is, no manual review). 

These types of applications have no established name. 

We are calling them automated decisionmaking applications. 

AUTO M AT E p_ D EC IS 10 SHAKING _APPL I CAT ION S 

Automated decisionmaking aool i cat ions are computer 
programs that initiate action (though output) on the basis 
of programmable decisionmaking criteria established by 
management and incorporated in computer instruct ion. The 
distinguishing characteristic of these application s> as cr - 
pared to other computer application, programs, is that marV 
of the computer's actions take place without manual revi&uJ 
and evaluation. 

An inventory application is an examole of a computer 
application program. If the computer processing of a requi- 
sition for material reduces the onhand quantity. below the 
.reorder point and if the computer issues a ourchase order 
without anyone reviewing the proposed procurement quant ity , 
then the application is an automated ■ decisionmaking appli- 
cation.’ Some of the computer output ■ of . these applications 
is reviewed. In the. foregoing example, the application 
may call for manual reviews of quantities on all purchase- 
orders over $5,000, with all purchase orders under that 
amount being released without review. 

• vie reviewed these applications because (1) billions 
of dollars are involved in the un reviewed actions that 
they initiate and (2) of indications that funds were being 
wasted because of incorrect actions. 
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CHARACTERISTICS 

One objective of. using computers operating under auto- 
mated decisionmaking applications is tc take advantage of 
their speed, accuracy, storage capabilities, and capacity 
to obey predetermined instructions. . These applications 
are needed in part, because of the tremendous volumes of 
information to be obtained, manipulated (processed), analyz- 
ed, and acted on in carrying out agency missions and goals. 

Automated decisionmaking applications process large 
volumes of transactions put into the computer system from 
various sources. They make repetitive decisions that, in 
many cases, previously have been made by people. The 
decision instuctions, built into the program, ask questions 
about the transactions and then initiate many actions throuan 
output. The actions depend solely on the criteria (logic) 
and data inside the computer system. 

Computer program 

The computer program (software), written by people, 
instructs the computer (1) to examine the input data and/ 
or data already in automatic data processing (ADP) files, 

(2) to perform logical decisionmaking steps 3nd computa- 
tions in processing the data, and (3) to initiate actions 
in the form of output as a result of this process. 

Input 

Data is usually obtained by people from various sources 
and is put into the computer in machine-readable form 
(including punched cards, optical character recognition 
documents, paper tape, magnetic ink character recognition 
documents, and direct keyboard entry). The data can be 
entered directly for processing or can be recorded on ADP 
files for processing at a later tinr. 

Output 

The application outputs are such things as (1) direc- 
tives to act (such as orders to ship material}, (2) payment 
authorizations or checks, (3) bills, and (4) notices. A 
large percentage of the output of these applications is not 
manually reviewed and evaluated, by people. 

The following illustration shows a. computer operating 
under automated decisionmaking applications. 
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Itie form of output varies (including listings, magnetic 
tapes, preprinted forms, and punched cards). These outputs 
indicate the decisions resulting from computer processing 
directed by the software. 


The outputs that are not reviewed or evaluated are 
usually issued to the organizations and people which take 
the action being directed or which are being paid, billed,, 
or notified. • • 

Some of the output of many automated decisionmaking 
applications is manually reviewed. Under •‘management by 
exception" principles, some output, the nature and extent 
of which is determined by management, is sent to people 1.1 
the organization for manual review and ^evaluation. This 
technique allows people to consider criteria, factors, and 
information not contained in the computer system in deciding 
whether the computer-directed action should be taken. For 
th:*se applications manual intervention takes place only for 
tlse actions output for review. 

The criteria for directing manual review of t*.e output 
are contained in the decisionmaking part of the program. In 
the inventory application example, the program would direct 
that purchases over $5,000 be output for manual review. The 
applications can be programed so that none of the output will 
be manually reviewed or evaluated before actions are taken. 

CONTRAST WITH OTHER COMPUTER APPLICATIONS 

Application programs designed to provide output to peo- 
ple for information and analysis are not automated decision- 
making applications. Many types of these application pro- 
grams ate used in Government, and the outputs are -^ent to 
people for review before actions are taken. 

Typical application programs that are not automated 
decisionmaking applications include: 

— Systems that make recommendations, all of which are 
manually reviewed before actions are taken. 

——Management and other information systems which pro- 
vide data to various revels of managers to assist 
them in making policy, management, and operating 
decisions. 


— Most mathematical models. 
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CHAPTER 2 

USE OF AUTOMATE D DECISIONMAKING APPLICATIONS 
BY FEDERAL AGENCIES 

_ Many Federal agencies use automated decisionmaking 
applications to support their functions. Annually, more than 
a billion actions, involving billions of dollars, in direc- 
tives to actj, to make payments, to issue orders for material, 
and t° bill for amounts owed are initiated. They also issue 
millions of notifications to people outside the Government. 

INFORMATION ABOUT AUTOMATED DE CISIONMAKING 
APPLICATIONS USED BY FEDERAL AGENCIES - 


We wanted to learn how these applications were used and 
t0 ^ ai! \ data on their characteristics and monetary impact 
or. Federal operations, but we found no central inventory. We 
developed a questionnaire to gather information 
f ih* F t d ?!; a i a “ toniat ed decisionmaking applications and dis- 
tributed it <.o 15 agencies that use computers extensively, 
the information :*e wanted included; ■ 


Functions supported by these applications. 

—Numbers of these applications and their impact on op- 
erations (including output produced and annual volume 
ard monetary impact), 

—Whether certain parts of the decisions were being man- 
ually reviewed. y 

We obtained store detailed information about selected 
automated decisionmaking applications to understand and il- 
lustrate typical uses. 


Almost all the agencies 
their automated decisionmaki 
is susimarized below. 


we contacted gave us examples of 
ng applications. The information 
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Defense 

departments 

and 

agencies 

Air Force 
Army 

Defense Supply 
Agency 

Navy 


Number 

of 

example s 

14 

14 

9 

18 


Civil 

departments 

and 

agencies 

Agriculture 
Commet ce 


Ad- 


General Services 
ministration 
Heal t'n. Education, 
and Welfare 
Housing and Urban 
Development. 
Interior . 
Transportation 
Treasury 

Railroad Retirement 
Board 

Veterans Administra 

tion 


Number 
. of 

examples 

6 

4 


5 
8 

6 

10 

18 

4 

3 

_9 

73 


Total 55 

Total number of examples obtained 1 jj8 
FUNCTI ONS SUPPORTED BY AUTOMATED 

deciTionmaking applications 

The Questionnaires showed that automated decisionmaking 
applications supported many functions. A compilation of re- 
sponses is presented below. 

Number of 
■ times function 
was cited 


Function 

Controlling 

notification 

Fiscal 

Payment 

Supply 

Billing 

Distribution 

Eligibility 


Number of 
times function 
was cited 

48 

48 

46 

46 

44 

41 

38 

31 


Function 

Maintenance 

Procurement 

Diagnostic 

Scheduling 

Disposal 

Cataloging 

Personnel 

Safety 


30 

30 

23 

20 

17 

13 

li 

9 


Dumber of a utomated decisionmaki ng 
applications an d their impact on 

FEDERAL AGENCIES 


No one collects statistics on these applications for 
the Federal Government as a whole, so we could not e ermine 
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the total number. Some of the agencies resoondinq to our 

chsit «*p™« con.«,t^ <* 

aSs report about automated 

• impact 

severa? e difffr2 S f S ^ identi f ied 128 'applications- which issued 
several different types of unreviewed 'out put . The nature nf 

,t „° Ut £ U L al ? d US annual impact on PederL opera- 

tions. both in volumes and dollars, are summariael teloS? 


Nature Oi. output 


Payment authorizations or 
checks to: 

Contractors or grantees 
Members of the public 
Government employees 
(other than pavroli ) 
Bills sent to: 

Contractors 

Government organizations 
Members of the public 
Purchase orders or supply 
requisitions 

Directives to ship material 
Directives to dispose of ma- 
terial 

Production, repair, -or rework 
schedules or instructions 
Notifications to members of 
the public 
Other 

Total 


Number' 

cited 


Total 

actions 


Total 

monetary 

impact 
— ■■ 


(000 ( 000,000 
omitted) omitted) 


10 

8,700 

$ 7,221 

23 

715,000 

18,589 

3 

200 

8 

3 

100 

15 

17 

17,300 

6,549 

18 

. 19,100 

3,298 

24 

28,000 

4> 456 

22 

. 260,200 

a/2,500 

11 

8,000 

a/56 

12 

191,300 

a/1,150 

21 

22,200 

N/A 

' 48 

447,300 

N/A 

212 

1,717,400 


i al on 

which these 

actions 


— *■ — kj j. iu cs ; 

were taken. Information rnat tne 

transportation costs represent about 5 Percent of the value' 
of material shipped: the. disposal costs^bout 3 pe r ^„t of 
the material disposed of; and production, repair or re- 
won cost about 23 percent of the value if Se mketi". 

ule *" 4 *;?««•/ impact in the preceding scbed- 

ule «.e for only a portion of the 212 out out tvpes. M-nv re- 

Cur n fn? 1 n 0 lcate ^. thac this data was not readily available. 

Cur followup confirmed this. ^ 
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REASONS FOR OUTPUT_OF .ACTIONS 
F 05 liANUAL_R§V I EW_AND_E VALUATION 

Some of the applications initiate all actions without 
review Most ate designed, however, under the mansgement-by- 
exceo?ion principle, which results in. some of the. output being 
reviewed by employees before the actions are implemented. 

Several reasons given by agencies for reviewing some of 
the output are shown below; 

Times cited 


Monetary value of indicated action exceeds 
prescribed dollar limitations 
Criticality of the action to be taken 
Eligibility factors related to the action 
Geographic considerations of various types 
Health and safety considerations related 
to the action 


43 

28 

21 

11 

10 


The percentage of actions initiated automatically varies 
from one Application to another and can bo adjusted by chang- 
ing the processing criteria. The percentage o£ uncevie e 
actions identified by agencies participating in this study 
is shown below. 


Percent of 
action s_un re v i awed 

100 

90 to 99 
80 to 89 
70 to 79 
60 to 69 
50 to 59 
Below 50 

No data provided 
Total 


Number of. applications 


35 

42 

13 

14 
5 
3 

14 

__2 

123 


AN EXAMPLE _OF WHAT _AUTOM.\rCD 
DECISIONMAKING _APPLICATICNS_DO 


make 


Automated decisionmaking applications are Designed to 
internal decisions o £ varying degrees of complexity and 
to generate output containing the action “ 
example of one of these applications is 
Other examples are presented in chapter 


to be taken. An 
town in this section 
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Customer returns program 


The Defense Supply Agency (DSA) uses an automated 
decisionmaking application-credit returns — to evaluate in- 
quiries from military activities on what to do 
plus DSA-managed material. The options are to 
the material for credit* (2) return it without 
(3) dispose of it. 


with sur- 
(I) return 
credit, or 


PSA's computers receive the requests in mach.ine-readabl 
form. The application identifies the commodity and refers t 
pertinent data about it from the ADP files (such as informa- 
tion on the quantities of the material already stored in 
PSA's inventory and expected future requirements). Using 
this and still other data, the application tells the activ- 
ity what to do with the material. usually these directives 
are sent without manual review. 


During a recent 1-year period, two of the six DSA sup- 
ply. centers issued the following unreviewed directives using 
this application. 


Nature of 
directives 


Ship the materia* (with or 
without credit) to the DSA 
supply system 
Dispose of the material 


Estimated volumes of 
unreviewed directives issue d 
Number of Val ue o7 

directives material 


174,000 $ 76,000,000 

62,0 00 24*000,000 


Total unreviewed advices 


236,000 $ 100,000,000 
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CHAPTER 3 

AUTOMATED DECISIONMAKING APPLICATIONS 
CAN MAKE BAD DECISIONS 

Whether actions initiated automatically by. the computer 
are correct or not largely depends on .(1.) the internal logic 
of the program and (2)' the .data that is fed into the system. 

Computers will produce bad decisions (1) if programers 
and analysts make misjudgments or errors in establishing 
the decisionmaking cr iter ia or (2 ) . i f the application is not 
designed and/or' coded in a manner that properly implements 
the decisionmaking criteria. .Changing circumstances can ma e 
adequate oaci sionraaking criteria in the software obsolete, 
and bad decisions will occur if the software is not changed. 
Failure to design appropriate checks on input data, such as 
edit checks, can contribute to bad decisions. These applica- 
tions can also make bad decisions if the data supplied to 
them is incomplete or incorrect or if the data is not ob- 
tained or processed quickly. 

Some internal audit groups have reported on bad deci- 
sions made by Government automated, decisionmaking applica- 
tions. The computer-initi rated actions caused the agencies 
to incur tens of millions of dollars of. unnecessary costs, 
premature costs, and overpayments. 

Such bad decisions may also harm individuals and impair 
an agency's ability to carry out its mission effectively. 

CONDITION S LEADING TO BAD DECISIO NS 

Adverse conditions common to several agencies have been, 
reported. These conditions, resulting in the applications 
automatically initiating uneconomical or otherwise incorrect 
actions, can be broadly categoried' as (1 ) software problems 
and (2) data problems. 

Software problems 

Several software problems that can cause bad decisions 
by automated decisionmaking applications include: 

— Designing software with- incomplete or erroneous deci- 
sionmaking criteria. Actions have been incorrect be- 
cause the decisionmaking logic omitted factors which 
should have been included. In other cases decision- 
making criteria included In the software were inappro- 
priate, either at the time of design or later, be- 
cause of changed circumstances. 
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--Failing to program the software as intended by the 
customer (user) or designer , resulting in logic 
errors often referred to as programing errors. 

— Omitting .seeded edit checks for determining complete- 
ness of input data. Critical d 3 ta elements have been 
left blank on many input documents, and because no 
checks were included, the applications processed the 
transaction with incomplete data. 

Data problems 

Input data, qual ity rs frequently a problem. Since much 
of this data is an integral part of the decisionmaking pro- 
cess, its poor quality can adversely affect the computer- 
directed actions. Problems includes 

— Incomplete data used by automated decisionmaking ap- 
plications. Some input documents prepared by people 
omitted entries in data elements which were critical 
to the application but which were processed anyway. 

The documents were not rejected when incomplete data 
was being used. In other instances data which the 
application needed and which should have become part 
of ADP files was not put into the system. 

• --Incorrect data used in automated decisionmaking ap- 
plication processing. People have often uninten- 
tionally introduced incorrect data into the ADP sys- 
tem. This incorrect data affected application deci- 
sions. 

— Obsolete data used in automated decisionmaking ap- 
plication processing. Data in the ADP files became 
obsolete due to new circumstances. The new data 
may have been available but was not put into the 
computer. 

Conditions that have been 
reported by internal audit 

Unfavorable conditions were identfied by 32 internal 
audit reports of 7 agencies. These reports, issued during 
a 23-month period, demonstrated that the same conditions 
occurred in different agencies and were therefore common 
problems. The audit reports, however, did not show the to- 
tal occurrences and dollar impact of these conditions, past 
or present, in federal automated decisionmaking applications. 
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The results of our analysis of those audit reports are 
summarized in the following table. (S' or further details, 
see app. V.) 

Number of 

' times 




Number of 

condition 


Number 

internal 

was 

Category and 

of 

audit 

reported 

condition 

agencies 

reports 

(note a) 

Software problems: 

Incomplete, erroneous 
or obsolete decision- 




making criteria 

7 

14 

30 

Programing errors 

5 

10 

10 

Criteria or programing 

(note b) 

5 

11 . • 

14 

Absence of needed edit 

checks 

4 

5 ■ 

11 

Data problems: 

Data elements incom- 

plete 

6 

10 

16 

Data elements incorrect 

5 

11 

30 

Data elements obsolete 

3 

. 3 

3 


a/Each condition can occur &ore than once. Software problems, 
~ such as programing errors, may have occurred in more than 
one portion of the program or the condition may have been 
observed at more than one location, each designing its own 
program.. The data conditions were based bn the number of 
different data elements that were either, incomplete, in- 
correct, or obsolete at least once. 

b/Internal audit reports were not sufficiently detailed to 
arrive at an opinion as to whether the problem was in cri- 
teria or programing. 

. Only 13 of the 32 reports had estimates of the monetarv 
impact of bad decisions, but these estimates ran to tens of 
millions a year in unnecessary and premature costs and in 
potential overpayments. Some reports cited specific cases 
but provided no estimates of the total monetary impact. 

Other reports cited potential mission impairment and possible 
harm to, individuals. 

The following sections are based on internal audit re- 
ports selected from the 32 reports obtained. 


12 


Approved For Release 2002/06/05 : CIA-RDP79-00498A0003001 10009-7 


Approved For Release 2002/06/05 : Cl A-RDR79-00498A0003001 1 0009-7 


S OFTWARE PRO BLEMS REPORTED 

Examples of software problems are presented to 
demonstrate the problems frequently experienc'd- with auto- 
mated decisionmaking. The examples ate not intended to be a 
criticise of the agencies involved, because these problems 
can ' occur wherever these applications are used. 

Army processing of requisitions 
for shipment to overseas locations 

Several Army inventory control points provide, material 
support to overseas customers which submit requisitions for 
materials to the control points. Automated decisionmaking 
applications are used to screen material availability at U.S. 
depots. The computer preduces a directive which is automat- 
ically issued to a depot to ship material to the overseas 
customer. These applications process over 100,000 overseas 
requisitions annually. 

Early in the 1970s the Army implemented a system de- 
signed to improve supply support to overseas customers from 
U.S. depots. The control points were instructed to design 
their ADP applications so that material would be issued from 
east coast depots to satisfy European customers and from west 
coast depots to satisfy Pacific customers. Controls were 
required to prevent the. software from releasing cross-country 
shipments without manual review. ' 

•The Army Audit Agency examined the applications in ef- 
fect at five control points. At four activities it found 
that the applications were not adequate to insure maximum 
filling of requisitions from the appropriate depots. For 
instance, in the initial requisition processing for overseas 
customers, the software used by one of the high- volume con- 
trol points screened stock availbility at eight depots be- 
fore finding the appropriate depot. For releasing back- 
ordeied-stock requisitions, depots on the opposite coast 
were often selected for material availability. The auditors 
reported that., at three control points, controls t'' prevent 
the automatic release of material from the wrong depots were 
not implemented and material was automat ical ly released for 
cross-country shipments. At least two control points used 
software that existed before the criteria for supporting 
overseas activities were developed. 

The audit agency estimated that, because of the use of 
this erroneous criteria, unnecessary transportation costs of 
$900,000 a year were incurred. In addition; $1.3 million 
was incurred in increased inventory investment (pipeline) 
costs. 
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The Army Materiel Command agreed with the audit agency's 
assessment of the problem and promised to revise the criteria 
contained in Army control point applications. 

Navy scheduling of aircraft 
equipment for overhaul 

The Navy's central manager for aircraft spare equipment 
and parts uses a computer to. identity and schedule overhaul 
for reparable components needed for future use. Until April 
1974 the application used was called the Navy integrated 
comprehensive reparable item scheduling program. 1/ 

This application considered inventory on hand, require- 
ments, and other data in ADP files to determine 

— which components should he scheduled for. overhaul, 

— what quantities should be overhauled, 

- — which depots should do the work, and 

— what priorities depots should give in deciding which 
items should be overhauled first. 

Depots used punched card output as the basis for sched- 
uling components for induction into their overhaul facil- 
ities. Priority levels shown on the output affected the de- 
pots' decisions regarding which items and quantities would 
be overhauled first. (Not all the quantities the program 
indicated for overhaul were processed because of limited de- 
pot overhaul capacity.) 

The priority levels shown in. the output ranged from 
level 0 (zero) — highest priority — -to level 3 — lowest prior- 
ity. 

During a 1-year period. 2/ Navy facilities spent about 
$145 million to overhaul aircraft components valued 


1/In April 1974 the Navy integrated comprehensive reparable 
™ item scheduling program was • replaced by another automated 
decisionmaking application called cyclical repair manage- 
ment. We believe that the problems that occurred m the 
first program could affect cyclical repair management m 
a similar way, but GAO's review did not evaluate the new 
program. 

2/The figures presented are for an overlapping but not iden- 
” tical period. The overlap is 6 months. 
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at about $797 million— mostly on the basis of the program's 
output. The Naval Audit Service, reviewing the operation, 
identified several major software problems, all of which 
resulted in overstating overhaul requirements. 

— A data element used in computing priority level 1 
contained data that resulted in duplications in com- 
puting levels 2 and 3. Gross overhaul requirements 
scheduled by the program were therefore overstated. 
When the program was designed, this duplication was 
overlooked. 

— Data elements showing recurring material usage, used 
to compute levels 2 and 3, were greatly overstated 
because of two software proble ms. 

1. Required reductions to the material usage quan- 
tities were not made automatically, because cer- 
tain Navy activities were leaving a data ele- 
ment blank on input documents sent to the cen- 
tral manager. Our followup determined that be- 
cause of the designer 1 s oversight or judgment 
error, no edit check was placed it the software 
to detect this missing data. 

2. There were no software procedures for automat- 
ically reducing recorded material usage quan- 
tities when customers canceled back orders and 
requisitions. Our followup disclosed that when 
this application was designed, the designee be- 
lieved that canceled back orders and requisi- 
tions would rarely occur. 

The Naval Audit Service estimated the effect of these 
incorrect actions was millions of dollars in unnecessary and 
premature overhaul costs. Although the Navy Command offi- 
cials did riot agree with the auditor's reported figures, 
they agreed that the problems identified were valid. Cor- 
rective actions have been taken or initiated. 

A GAO report (B-I62J52, May 21, 1974) “Better Methods 
Needed for Canceling Orders for Materiel No Longer Required' 1 
discussed the Navy's practice of not automatically reducing 
recorded material usage when unfilled customer orders were 
canceled. The report stated that “we estimate that this 
overstatement resulted in annual unnecessary materiel buys 
and repairs totaling about $1C million." Cf that amount, 
more than $3 million was for repairs initiated by this auto- 
mated decisionmaking application. 
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DATA PROB LEMS REPORTED 

The following examples of data problems show how bad 
data can adversely affect the actions directed hy automated 
decisionmaking applications. ■ .... 

Veterans Administration payments, for . , 

apprent iceship and other on-j ob training 

The Veterans Administration (VA) uses a computer appli- 
cation to make monthly payments to more than 185,000 veter- 
ans in apprenticeship or other on-j.ob training. This appli- 
cation is designed to make payments at a rate that decreases 
every S months, under the assumption that veteran’s pay will 
increase a- he learns his trade. 

. put into the computer is the basis for automati- 
cally a- • mining the rates at which the veteran will be 
paid. _..„»» month, additional data is put in regarding the 
veteran's continuing eligibility to receive the payments. 

This application is programed to read input documents 
and distinguish apprenticeship and other, on-job training 
awards from other types of education awards. When the appli- 
cation recognises these on-job training awards, it refers to 
appropriate rate tables to determine the. proper payment. 

The application refers to a new lower rate every 6 months 
and automatically initiates payments, at the reduced rate. 
Annually, this application, initiates about 1.4 million un- 
reviewed checks for more than $225 million in apprenticeship 
and other on-job training awards. . : 

Two types of input documents initiate payments for 
these awards. An original award document is designed to 
initiate payments to a veteran not previously receiving them. 
If the veteran has already received benefits and there is a 
need for (1) reentrance, (2) a supplemental award, or (3) new 
key data such as dependency changes, a different input docu- 
ment ( supplemental award, code sheet V is prepared. Both 
documents contain data elements that allow the computer to 
determine that it is an apprenticeship and other on- job 
.training award and that the reducing rate taole should be 
used. 

The data entry on the supplemental a'waid document that 
causes the program to build the scheduled rate reduction is 
code 77 in a data element called change reason. 

VA internal auditors reported that 22 of 121 tested 
supplemental award documents for these benefits did not 
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contain change reason code 77 on. the input documents (the 
data problem). These documents were received from 10 dif- 
ferent VA locations. The application accepted and processed 
tne documents fc -cause the software did not contain ar. edit 
check to disclose and reject documents ■ with incomplete en- 
tries in this data clement (a related software problem! . 

Because the data was incomplete, the computer used a 
single rate for the entile period of training at the highest 
step indicated. This problem caused potential overpayments 
of $700,000. . 


Possible causes cited for processing incomplete input 
documents included new pet sonnel — requiring additional 
training and fatigue. The designer over looked the needed 
edit check, a software problem, in preparing the detailed 
and complex software. 


Army proce ssing requisitions' 
for radioa ct ive* ma te; i~a l 

The Army uses a computer to automatically process cus- 
tomer requisitions fot commodities. One As my agency u*os an 
application to process at least 250,000 requisitions annually 
tor material valued at a minimum of $250 million. About 35 
percent of the customer requisitions are output foi manual 
review and evaluation foi any of sevei al reasons. The te- 
mainang 65 percent ai e processed without manual ic-view. 

Some commodities the agency manages contain r adioacti v« 
material. The Army master data ADP file is supposed to con- 
tain a special control code (code 3) in a specific data ele- 
ment for commodities containing radioactive material. This 
code, which should be put in bv item managers, prevents 
automatic issues. The item managers receive commodity req- 
uisitions for review and evaluation. This manual interven- 
tion is required to insure that the requi si t i oner s ate (1) 
author i zed to receive the material, (2) aware of the radio- 
active content, and (3) aware of the safeguards that must be 
used. 


ihe At rr.y Audit Agency reviewed 86 radioactive commod- 
ities which t ho agency managed to determine if the ctopet 
special control item codes wti e contained irt A BP files. The 
review showed that 29 of the commodities wet* incor : ectlv 
coded. " 


Eleven commodities were coded as 
(code 1) but not as radioactive, 
is one that is scarce, costly, ot 


a regulated item 
(A regulated item 
highly technical.) 
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’ — Eighteen commodities contained an 0 code in the ADP 
files. An 0 code indicates that no special controls 
or handling are required. Many requisitions for 
these commodities, are processed automatically. 

Most of the incorrectly coded commodities had b.en in the 
supply system 4 to 11 years. 

During the Audit Agency’s review of 1 year's transac- 
tions, at least 33. customer requisitions were automatically 
filled for 18 incorrectly coded commodities. Army customers 
and foreign governments under military- assistance programs 
were issued 423 units on these 38 requisitions. 

Since the commodities were incorrectly coded, the item 
managers did not coordinate the issue of the units with the 
38 customers. Consequently, there was doubt that the cus- 
tomers should have been issued the material or that they 
were aware of the radioactivity in the commodities. 

Army officials cited the following possible reasons for 
the incorrect codes contained in APP. files. 

— The item managers who prepared the input to ADP files 
may not have been fully aware of the requirements and 
procedures for coding radioactive material. 

--The agency's health physicist may not have notified 
the item managers of the radioactivity contained in 
these commodities. 

— The item managers may have been notified but failed 
to input the correct data codes. 

Army officials agreed with the Audit Agency's findings 
and said they would (1) correct the ADP files for all radio- 
active commodities, (2) reemphasize to item managet s the 
need for assigning the proper special control item code to 
commodities, and (3) have a heal th • physicist study the com- 
modities to insure that .the items could be used safely by 
the customers that received them automatically. Toe special 
study determined that the commodities involved could be 
safely used by the recipients. 

INTERNAL AUDITS OF AUTOMATED 
DE CISIONMAKING APPLICATIONS . 

Since published internal audit .reports were the sources 
of our information on bad decisions, we asked nine internal 
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audit groups about the nature, approaches, and frequency of 
scheduling audits of these applications. 

We learned that certain internal audit groups rarely 
became involved in the applications* logic because they 
lacked the expertise to effectively make such studies. 

No internal audit group has prepared lists of agency- 
automated decisionmaking applications and scheduled reviews 
fche.tr decisions, either routinely or when the system is 
modified. However, several audit groups schedule specific 
agency functions for audit,' and if the functions are sup- 
ported by these applications, auditors will get involved in 
the internal decisionmaking logic to evaluate the agency's 
performance. . . . • 

Agency functions are generally audited on a cyclical 
basis, but the cycle may be anywhere from 2 to 8 years. 
Ordinarily, the frequency of review is not dependent on 
whether the function is supported by an automated decision- 
making application. In addition, auditors may review func- 
tions and related automated decisionmaking if there is (1) a 
special request or (2) an indication of a problem based on 
complaints. On the basis of approaches, taken by internal 
audit groups, it appears that many of these applications go 
unaudited for long periods of time or may never be audited. 

Although many of the audit reports adequately show many 
of the common problems that exist, they do not show the 
overall impact of the problems for all automated decision- 
making applications. In fact, there is no basis for estimat 
ing the total impact of bad decisions currently being made 
by these appl icat ions. " 
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CHAPTER 4 


‘ CAUSES O F BAD A UTOMATE D DECISIONS 

The two basic automated decisionmaking application 
problems, software and data, are often interdependent. For 
example/, automated deci sionmaki na applications making bad 
decisions because of incomplete data elements on input docu- 
ments illustrate both a data o».oalem and a software problem 
because (1) input documents h not been properly prepared 
(data) and (2) edit checks for completeness have not been 
properly designed (software). Other problems, such as when 
incomplete or erroneous decisionmaking criteria are used 
(software) and incorrect data is put into the application 
(data}, can occur independent ly. 

The problems in each of these two areas are caused by a 
variety of factors. We identified many causes of these 
problems by (1) corresponding with people experienced in 
software design and data management, (2) discussing them 
with officials of selected Federal agencies, and (3) ana- 
lyzing published internal audit reports. 

SOFTW ARE PROBLEMS 

Computer programs are usually developed and modified 
by a combination of people: the user (or customer), that 
requires the computer assistance; the designer (or analyst), 
who translates the requirements of tne user into a logical 
structure; and the programer, who translates the logic into 
program instructions which can be recognized and used by the 
computer . 

The software development and modification process was 
similar at each Federal agency we visited. Variations are 
not related to the process itself but .rather involve such 
factors as 

— organizational setup and physical locations; 

—titles of people performing various aspects of the 
work? and 

— nature of the documentation that will be prepared, 
such as use of program flow charts. 

Causes of software problems 

Agency officials said that the design or. modification 
and programing of software could not be guided by specific 
instructions on how best to do the work. Instead, agencies 
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rely on people who know (l) the function supported by the 
computer and (2) the art of design and. coding so that the 
computer can perform the desired tasks. Some agencies pro- 
vide broad guidelines on the process, the documents to be 
used in the process (documentation), and at one agency, 
instructions on whet designers and programers snould con- 
sider when doing the work. . ' - 

The user initiating the work sets- forth many of the 
specifics regarding the internal decisionmaking criteria 
to be used. Often the designer makes some decisions. Both 
act on the basis of their knowledge of the function, avail- 
able guidelines in terms, of ' management instruction or legis- 
lation, their perceptions of the transactions to be process- 
ed , and communications with each other. Sometimes they will 
call on operations research experts to help them design new 
criteria, rfhile sometimes they will use - existing criteria 
to process similar transactions. 

The designer takes the established criteria and pre- 
pares moce specific documentation which is used for program- 
ing. The design and programing documents developed become 
very detailed and complex, because the computer is instruct- 
ed to operate in a logical step-by-step manner on a large 
number of different conditions. Even less complex appli- 
cations can consist of thousands of individual instructions 
that must be designed and programed to do what the user 
and designer perceive to be correct. . 

The designer and user are usually responsible for 
designing edit checks into the program. This includes 
checks for the' completeness of data elements on input 
documents. According to agency officials, edit checks are 
placed in the software for data that is critical to the 
decisionmaking, such as when incomplete or erroneous data 
can affect the determinations made by the computer. Some 
officials said that edit checks are placed for almost every 
data element. One agency is making an overt effort to 
limit edit checks to reduce the number of documents reject- 
ed by the computer. 

In developing software, it is generally accepted that 
the lines of communication between the user and designer 
and the designer and orogramer must, be effective. 

To identify some' of the - causes for the software prob- 
lems presented in chapter 3, we' 

— discussed them with Federal officials at several 
agencies; 
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--received responses to questionnaires from 257 

individuals who are experienced in the areas of ADP 
software design, modification, and programing; and 

— analyzed causes cited by internal auditors. 

A schedule summarizing some of the causes of software 
problems is followed by a discussion of each. 


Summary of Ca uses of. S oft ware Problems 


Opinions of people Identified from 


answering the 

quest ; anna ice — 

contacts 


deg 

ree of cause (note a) . 

with officials 

Cited as a 


Moderate 

■Somewhat 

of Federal 

cause by 


to 

small. 

agencies 

inter nal 

Cause very large 

or none 

(note b> 

auditors 

Inadequate 





communications 
between the parties 
to software design 

251 

4 

X 


Incorrect perceptions 





of the nature of 
actual transactions 
to be processed 

213 

22 

X 

X 

Inadequate documentation 





preventing adequate 
reviews of software 

225 

28 

X 

X 

Time constraints 





hampering the effec- 
tiveness of the 
design process 

216 

4° 

X 


Absence of written 





criteria or guidelines 
for designers to follow 

?v)4 

49 

X 


Detail and complexity 





involved in designing, 
coding, and reviewing 
software 

177 

70 

X 

X 

Reliance on the expertise 





and experience of 
people doing the work 
(state of the art) 

171 

8-3 

X 

X 

Undetected changes in 





circumstances making 
the application obsolete 

167 

90 

. X 

X 

State of the art in 





software testing which 
prevents testing all 
possible conditions 

16 4 

51 

X 



a/The questionnaire presented "some possible causes of tne design conditions 
~ (problems) * * and asked that "Eased, cn your software desig experience 

* *'* indicate the degree to which you believe each of these causes contrib- 
utes to the design condition ‘problems) general." The? responses allowed 
were to a: very large degree, somewhat large degiee, moderate degree, some- 
what small degree, very small degree, or rot at all. 

b/Our contacts were made with various organizational elements, excluding inter- 
— nal aod it, within five agencies: Department of the Navy; Department of the 
Air Force? Department of Health, Education, and Welfare; Veterans Administra- 
tion? and National Bureau of Standards. 
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The problems identified are caused at various phases 
of the software design process including 

* — user determinations, 

“designer actions, and 

— program coding. 

Many problems are not detected through the review and test 
phases of the process and are therefore continued through 
implementation and operation of the automated decisionmaking 
application. Officials at the National Bureau of Standards 
and the Air Force believe that it is impossible to insure the 
design of completely error-free software under the current 
state of the art. 

Inadequate co mmunica tio n between 
the parties to software design 

At least three groups of people must adequately com- 
municate to develop or modify the applications successfully. 
Assuming that the user knows what he wants the compute! 
to do and that his criteria are correct, inadequate communi- 
cations of this information can result in developing soft- 
ware that is not exactly what the user wants. 

Much has been written about the ' communication' problem 
in software development, and it is generally recognized as 
a human problem. 

I ncorrect perceptions of the nature 
o f actual transactions to be proces sed 

Decisionmaking criteria used in these applications have 
sometimes been erroneous, because people developing them made 
wrong assumptions about the nature of the transactions that 
were to be processed. They may have relied on limited data 
about the transactions and established the criteria on their 
judgment. 

Officials of one agency believed that a large percent- 
age of automated deci sionmaki ng application software prob- 
lems were caused at the very beginning of the design 
process by people involved in defining . requirements and 
establishing decisionmaking criteria. 

In other, cases, the designer may have used criteria 
contained in existing software to - process transactions in a 
similar, but not identical, environment. Sometimes this is 
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done to shorten design and programing time, but it can and 
has caused problems. 

Inadequate documentation, preventing 
adequ at e revi e ws of softw are 

In our October 8, 1974, report (B--115369) "Improvement 
Needed in Documenting Computer Systems," we noted that some 
agencies had not developed adequate guidelines for prepar- 
ing good documentation. Several Federal officials said 
that this was still a problem and that, documentation for 
many computer applications (including automated decision- 
making ) was inadequate. ; . 

The report stated: 

“In one case documentation explaining the objec- 
tives of the computer system was not prepared by 
the systems analyst. Without this information, 
management . could not adequately monitor the 
system's development. * * * the system did not 
accomplish the results originally intended. by 
management. 

"In another case, inadequate documentation was 
cited as causing management to . spend over a year 
to determine how the various programs in a 
complex system operated." 

Adequate design documentation is needed to allow fcr 

— reviewing the work done during application design and 
modification, - 

— making the necessary modification, 

—correcting errors when they are detected, and 

— insuring the application is operating as intended. 

Time constraints hampering the 
e ffectiven e ss of the d esign process 

rieny systems containing these applications are design- 
ed or modified because of legislation or other high-prior- 
ity requirements imposed by top management. Often this 
calls for implementation by a specific date. Developing 
and/or modifying software within the required time frames 
can hamper efforts to insure its adequacy. Agencies that 
must make changes to .hese applications on the basis of 
legislation include VA and the Department of Health, 
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Education # and Welfare (HEW). The Department of Defease 
often must make software modi ficat ions based on high-prior- 
ity requirements imposed by top management. 

Absence of w ritten cri teri a or 
g uidelines for designer's to follow 

Federal officials had many opinions about the need for 
and nature of written guidelines that should be provided to 
designers of software. The agencies we visited had varying 
degrees of formal guidelines, but none provided instructions 
on how to do design work. 

Some officials who believe that written criteria and 
guidelines on how to design software are not desirable refer 
to the process as an art that cannot be guided or improved 
by written instructions. However, the consensus of responses 
to our questionnaires indicates that the absence of criteria 
or guidelines can be a major cause for some automated deci- 
sionmaking application problems. 

Detail and complexity involv ed in 
designing, codin g, and re viewin g softw are 

Even smaller applications can be extremely complex and 
detailed when designing and coding the processing logic and 
edit checks. The complexities and detail involved may also 
hamper. the review process that may exist. 

An illustration of the problem is VA's automated deci- 
sionmaking application for supplemental education benefit 
awards — which is a small part of VA’s total education 
applications. This program consists of more than 1,100 
lines of code covering about 420 decision points. One Navy 
automated discosal application — also a relatively minor 
program compared to others — contains about 7,300 lines of 
code with more than 290 decision points. More complex 
software, such as the Navy cyclical repair management pro- 
gram, has more than 64,800 lines of code with at least 630 
decision points. 

The sheer detail and complexity of the process can 
cause design and programing errors and omissions which are 
not caught in review and testing.’ Therefore, bad decisions 
occur . 
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Relian c e o n the expertise and 
experience o f peo ple doing the wor k 

The nature of the design process. causes agencies to 
rely oh designers who must be experienced in both the 
software design and the function to be supported by. these 
'applications. 

Federal designers are 

— schooled in the art of software design and learn 
the function to be supported, 

— experienced in an operating function and learn 
the art of software design, or 

— former programmers and are promoted to the design 
function. Programers are generally schooled in 
writing code in specific computer languages. 

Much reliance is placed on the individual designer's 
ability to convert user requirements to the type of detail- 
ed logic needed for programer coding. Reliance is also 
placed on the programmer's ability to write code according 
to the logic given him. Because of the detail and complex- 
ity involved, it is difficult for management to review and 
assess every aspect of the designers '. and programers' work. 

Undetected cha nges in circumstances 
making the application obsolete 

A cause for erroneous decisionmaking criteria includes 
the failure to identify and/or to relate changes in pro- 
cessing circumstances to the operation of the application. 
Once the application is operational, it will make decisions 
1 — good or bad — on the same basis until it is modified. 

Not recognizing changed circumstances so that appli- 
cations could be modified could result in bad decisions 
based on criteria that were correct when designed, but 
which no longer applied. 

State of the art of rtogram 
testi ng which pi e verLs testi ng 
all pcssibie conditions 

The current stat® of the art makes it difficult for 
agencies to test for all conditions that may occur during 
the transaction processing. Most ayencies cannot even 
be sure that the tests nave exercised every line of code. 

As result, accepted software can contain design and/or 


26 


L • 

Approved For Release 2002/06/05 : CIA-RDP79-00498A0003001 10009-7 


rJ ■ 



Approved For Release 2002/06/05 : CIA-RDP79-00498A0003001 10009-7 


coding errors not identified during the test phase. Some 
of these errors may not be detected until long after the 
application beco.-.rs operational. 

The. inability to test for all conditions also pre- 
cludes a full evaluation of user and designer criteria 
built into the program (if and when such evaluation is 
attempted. ) . . . 

DATA PROBLEMS 

Data used by the computer in making decisions comes 
from a variety of sources, both internal and external to 
the agency that has the computer. A tabulation of the 
various sources of data input for the 128 automated deci- 
sionmaking. applications identified is presented below. 

Number of applications 

• , . • in which the originator 

Source of input document was cited 


People within the agency 

operating the application 49 

People located outside the 
agency operating the appli- 
cation but within the same 
Federal department or 

independent agency • 23 

People located in non-Govern- 

ment activities . 12 

People located in other Federal 
departments or independent - 

agencies . 7 

Control over the completeness, accuracy, and currency 
of data largely depends on the source. Obviously, the cor- 
rectness of an application operated at an agency where all 
the data comes from outside sources largely depends on the 
quality of data submitted.. Some controls can be applied to 
incoming, input, but they cannot guarantee completely to rror — 
free data. 

According to some Federal officials, the largest 
single data problem is validating input data. However, 
data quality must be controlled from" the moment data enters 
the system until the automatic processino is complete* 
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Types of controls for data 

There are two basic types of controls for insuring 
the completeness, accuracy, and currency of data used by a 
computer in making decisions. 

1. External controls are procedures developed outside 
the computer system. The objective is to check the 
quality of data to be put into and contained in the 
computer system/ The controls, include such things 
as manual procedures designed to determine if data 
is recorded completely and accurately on input doc- 
uments and whether documents are. being received 
and/or processed on time.' 

2. Internal controls generally do not involve human 
intervention. Many of these controls are ouilt 
into the software. They include edit checks for 
completeness, logical relationship tests (does the 
data make sense?) and reasonableness, checks (to 
isolate predetermined out-of-bounds conditions). 

According to the National Archives and Records Service, 
General Services Administration (GSA), both types of con- 
trols are necessary r 1 no automated decisionmaking applica- 
tion can be reliable, if either type of control is deficient. 

These- applications use data originally prepared by 
people. The data input process often consists of people 

— filling out hard copy documents, 1/ often on prede- 
signed standard forms, and 

— converting the data to a form that can be read 
by the computer — machine-readable form. 

As part of the external controls that should exist, 
the people doing the work should be qualified and adequate- 
ly trained. Adequate guidelines should be given to these 
people on a timely basis instructing them how to fill 
out the documents involved, including what entries should 
be made under varying circumstances. The forms (hard copy 
and input) should be designed to be as simple as possible 


1/Under some circumstances, such as source data automation 
and direct input devices, hard copy documents are not 
prepared . 
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to allow for easy reading by people. Procedures should 
exist for reviewing {i. e., statistical sampling) input 
documents to test their completeness and accuracy. 

Controls should also provide for timely processing of 
the data. . J 

If incomplete or inaccurate data enters the computer 
system undetected* automatic actions can be incorrect. The 
actions will continue to be incorrect if that data is 
stored in ADP files and reused. These applications can 
also make incorrect decisions if current data is not cut 
into the system. 

Causes of data problems 

To identify some of the causes ti the data problems, 
we 


contacted Federal officials at several agencies, 

— received responses to questionnaires from 205 
individuals who are experienced in the area of 
data management in computers, and 

analyzed causes cited by internal auditors. 

A schedule summarizing some of the causes of data 
problems is followed by a discussion of each. 
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Cause 


S uamary of Causes of Data Problems 

Opinions of people Identified froa 

answering the questionnaire contacts 

—degre e of causes (note a) with officials Cited as 
Moderate Somewhat of Federal cause by 

to small agencies internal 

very large or none ’ ( note b ) aud t tor s 


Forms designed and used 

for input preparation are 

too complex- i®3 


ADP files are not always 
adequately reviewed to 
assure that good data *s 
being used. I 7 ® 


Instructions to people 
preparing data input ate 
not always provided, are 
provided late* or are not 
adequate. 

Preparers of data input 
are not always adequately 
trained. 159 



x 


x 


Manual reviews of input 
documents are not always 

adequate. 144 61 x 

High volumes of transactions 
cause input preparers to 

make errors (workload .. . . 

pressure). 131 T* x 

a/TIte questionnaire presented "soiree possible causes of the data conditions 
- ( oroblems) * * * * and asked that ^based on your data management experi- 
ence * * * indicate the degree to which you believe each of these causes 
contributes to the data condition (problems) in general.* The responses 
allowed were to a: very large degree.- somewhat large degree, moderate 
degree, somewhat small degree, or very small degree, or not at all. 

b/Our contacts were made with various organizational elements, excluding 
~ internal audit, within six agencies: the Department of the Navy; De- 
partment of Health, Education, and Welfare; Veterans Administration; 
National' Bureau .of Standards; National Archives and Records Service; 
and Civil SeLvice Coiwsission. * 
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The errors occur at the source of data preparation. 

They are not detected by the various internal controls in 
the software' because .controls' for the specific error (1) 
are not. designed or (2) cannot be designed/ 

F orms designed and used for 
i nput preparation at e^TocT cornplex 

Using simple forms to record., . collect , transmit, and 
process information for input to computers improves the 
completeness and accuracy of the data eventual ly . used by 
all computer application programs. . The mere complex the 
forms are, the. more prone they are tO' data errors,., which 
can affect the correctness of actions initiated by auto- 
mated deci si oitrnak i ng app] i cat ions . 

A DP fi le s ar e not always 
a dequately rev iewed ~to~assur e 
tha t good dat a is b e i n q~u sed~ 

A recognized external control technique is to output 
and review data contained in ADP files. Failure' to do this 
can result in obsolete of otherwise incorrect data used in 
automated decisionmaking appl i cat ions. Incorrect deci sions 

a'~e therefore initiated. Without reviews, it is possible 
for some data errors to regain undetected for vears and to 
allow for an accumulation of errors compounding the problem. 

Instructions to people preparing data 
input are not always provided 7 ~a7e 
provid e d late , or ar e no t adequate 

It is important to provide clear instructions to people 
preparing input documents. Timely updating. of these instruct 
tions when chanqes occur is also important. The. failure to 
issue clear and timely instructions can cause data errors 
that may not be detected by internal controls.. 

Preparer s of data i nput, are not 
always adequate ly tVafned 

Most training in the input data preparation area is 
done . by individual agencies, because it , rr.u ;t be geared 
toward the individual application, each with its own 
special forms, data content, and related input media. 

Inadequate training, of persons involved in processing 
data to the computer ( suen as filling out forms and 
punching cards' cars lead to high error rates which result 
in bad decisions made by these appl ica*, ions. 
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Manual reviews of input documents 
are not always~aBequate 

External controls include selective manual reviews of 
input documents to determine completeness and accuracy. 

These reviews, made by supervisors or quality c. atroi groups, 
should be geared toward measuring the quality of data 
entering the system, including determining trends, signi- 
ficance, and sources of errors. 

When there are different types and sources of input, 
review procedures should cover. them all. Developing and 
monitoring statistical error rates is important. The review 
procedure, however, should also include determining the 
errors' potential materiality so that . management can make 
judgments on where corrective actions should be taken. 


Manual reviews supplement internal controls by (1) 
disclosing needed software data validation (such as edit 
checks) that is missed because of software problems or 
(2) identifying trends of material data errors which are 
not detected by software data validation. 

Bigh v olumes of trans act ions 
caused input preparers to make 
errors (workload pressures ) 

Automated decisionmaking applications ate designed, 
in part, to help organizations cope with the high volumes 
of transactions that have to be processed; Although the 
computer processes the data once it is entered, the volumes 
of documents (hard copy and machine readable.) that must be 
prepared are tremendous. For example, we estimated that 
during a 12-aonth period, the VA Center, Philadelphia, 
Pennsylvania, prepared more than 4 million documents for 
input to computers. Other VA activities throughout the 
Uni ted States al so prepare such input documents. The 
Navy Aviation Supply Office (ASO), also in Philadelphia; 
annually receives about 10 million transaction reports 
for input to computers. The transaction reports are 
mainly prepared by Navy facilities that receive, store, 
and issue aeronautical equipment. 

The volumes of data that must be processed by people 
recording material on original documents and preparing 
machine-readable documents can lead to workload pres- 
sures that result in data errors. 
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CHAPTER 5 

FEDERAL MANAGEMENT OF 
AUTOMATED DECISIONMAKING APPLICATIONS 

Although we believe that most decisions made by these 
applications are correct, we know from audit reports we re- 
viewed that they also make bed decisions that cost the Gov- 
ernment many millions of dollars annually. Additionally, 
bad decisions can impede agency mission achievement and may 
result in harm to people. 

To a large degree. software design and data quality con- 
trol are an art. Much of the process is imperfect because 
people instruct the computer and supply data to it. 

The fact that computers will act only as instructed by 
people, and on data prepared by people, makes them particu- 
larly susceptible to incorrect output, which in an automated 
decisionmaking application causes incorrect actions. 

Undetected errors in preparing the software— whether 
caused by the user, the designer, or the programer — car. 
cause the computer to repeat bad decisions. These errors 
will continue until the problem is detected and corrected. 

Data problems may be random or repetitive. The repeti- 
tive problems resulting from sue? items as inadequate in- 
structions and complexity of forms will also continue until 
corrective actions are taken. 

RESPONSIBILITIES FOR ADP MANAGEMENT 
IN THE GOVERNMENT 

Public Law 89-306, the Brooks Act, specifies the major 
ADP management responsibilities of the Office of Management 
and Budget (0MB), the General Services Administration (GSA), 
and the Department of Commerce. 

Under this act, the Administrator of General Services ... 
is charged with economic and efficient purchase, lease, and 
maintenance of ADP equipment by Federal agencies. The Ad- 
ministrator also has some control over using ADP equipment. 
The Department of Commerce is authorized to provide scien- 
tific and technological services for ADP systems and to 
make recommendations concerning ADP standards. Thi s is 
carried out through the National Bureau of Standards* Insti- 
tute for Computer Sciences and Technology. The act states 
that the authority granted to the Administrator of General 
Services and to the Secretary of Commerce is subject to 
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policy and fiscal control by 0M3. This constitutes 
oversight responsibility for the area. 

In response to Government needs for training and 
education in ADP, the Civil Service Commission's Bureau of 
Training •operates an ADP Management Training Center. 

This center offers a variety of courses to Federal civil 
and military personnel. Certain portions of their curriculum 
address the controls area in automated systems. The material 
presented should assist in alerting managers who take these 
courses to possible control weaknesses in their agency's 
operations. 

No Federal-wide guidelines on automated 
decisionmaking application s 

Neither GSA nor the Secretary of Commerce has considered 
these applications as a separate subject, matter for management 
consideration. There are, therefore, no established Federal 
guidelines for identifying, developing, operating, or moni- 
toring these applications to insure that, they ate operating 
effectively and economically. 

POLICY ACTIONS BY FEDERAL AGENCIES TO MANAGE 
AUTOMATED DECISIONMAKING APPLICATIONS 


No Federal agencies we contacted had considered these 
applications separately from other types of computer appli- 
cation programs in issuing management instructions. When 
instructions on software design had been issued, they were 
general and dealt with such things as 

--levels of approval required to initiate end process 
a design project; 

--concepts of project management — including setting 
priorities, establishing target, dates, and 
requir ing : cost-benef i t studies; 

— the phases of software design and the documentation 
required; and 

—testing and certification requirements. 

Considering the current state of the art and the human 
problems that exist, we agree with those Federal . of ficials 
who contend that issuing detailed instructions on how to 
design these applications (or other computer application 
programs) will not in itself materially reduce many of the 
errors that are made in them. . 
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Inv e nto r ies of automated decisionmaking 
appi icat'ions 

Agencies have done little to establish centralized 
information on computer .appi icat ion programs 'hat identifies 
these applications and shows their characteristics. 
Characteristics include the (1) nature of actions initiated, 
(2) monetary and other impact on operations, and (3) nature 
and sources of input. Information is' sometimes available 
within an agency but must be pulled together from different, 
sources. This is done mainly when requested by higher level 
sources, such as headquarters, a budget committee, or an 
agency such as GAO. It is. not. normally done. 

WHAT AGENCIES DO 

Ke studied what Federal agencies do in designing, 
modifying, testing, and operating these applications. We 
also studied how these agencies manage data entered and 
contained in their computers. The studies were made at 
selected agencies of the Department of Defense (Navy), HEW 
(Social Security Administration), and VA (education and 
insurance applications). We also visited a responsible 
headquarters agency in the Department of the Air Force to 
discuss these subjects on a limited basis. 

We examined policy and existing procedures and 
practices for managing computer application programs but did 
not verify that they were being employed as described to us. 

Despite the apparent' variances in the nature and types 
of policies and instructions issued, the same types of 
problems exist at these and other agencies. 

Design and modification 

VA had no written instructions for designing or modify- 
ing computer application prograr.^.- VA told us that it relied 
on written text material as a guide. VA has issued instruc- 
tions on establishing and controlling software design proj- 
ects, establishing approval levels, and establishing prior- 
ities and target daces. 

The Social Security Administration (SSA) has issued a 
guide that describes the various phases of the design and 
modification processes, establishes review and approval 
steps, and describes who is responsible for doing the work. 

Neither agei.cy has issued instructions on how to do the 
design work or what to consider when doing such work. VA 
officials do not believe that it is necessary or even 
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feasible to. issue such instruct ions. SSA assumes that 
designers and programers ate adequately trained ar#d experi- 
enced since courses are cont inual ly of fered so that skills 
can be maintained at a satisfactory level. 

The Navy Fleet Material Support Office (F21SOJ is the 
central design activity for Naval Supply Systems Command ac- 
tivities. They have issued instruct ions to designers and 
programers in the form of information processing standards. 
The instructions provide guidance on what, designers and pro- 
gramed ate supposed to consider when doing the work, in- 
cluding 

— customer and mandated requ i renent s ; 

— logical sequencing of ADP actions; 

— types of input and output; 

— data formats and uses; 

— data accuracy, completeness, and. currency require- 
ments; 

— error and exception conditions (edit checks); and 

— data volumes and frequencies. 

Independ e nt reviews of designed 
and modified produc t 

.The reviews of the detailed designed product 1/ are 
generally made by t he user and/or the people doing the work. 
According to agency officials the extent of these reviews 
varies from 

— a page-by-page analysis made by ASQ of products de- 
signed by FMSO to 

— a less formalized cursory review made' by. supervisors 
or management. 


i 


1 /Usual ly consist inq of a narrative oi flow-chart descrip— 

— tion of t he processing to be followed by the co-puter dur- 
ing ope rat i on . 
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lv T e observed no -requirements for making independent re- 
views of the detailed designed product, Essentially, the 
people doing the work are responsible for- doing the. detailed 
reviews, 

' ■ The Mr Force Audit Agency independently reviews se- 
lected -data processing systems before they ate implemented 
(preimplementation reviews}* These reviews, made at four 
Air Force design activities* include evaluating the designed 
computer application programs and related edit checks. 

This approach requires the auditor to become familiar 
with functions supported by applications , as well as learn- 
ing basic, soft ware design and data control concepts. It in- 
cludes reviewing and evaluating (1) the decisionmaking cri- 
teria, (2) the program coding, (3) the edit checks, and (4) 
other potential data problems. 

The Audit Agency had never calculated the cost savings 
that resulted from identifying and correct ing ’potent ial 
problems before the applications were placed into operation, 

A major reason cited was that since corrective actions were 
often taken on the spot, there was no need for estimating 
unnecessary costs that woula otherwise have resulted during 
operation. 

Preimplementation audit reports of the Air Force Audit 
Agency showed that many of the problems that, had been re- 
ported in operational automated decisionmaking applications 
were identified during preimplementation reviews, and Air 
Force design officials agreed that the problems existed. 

For instance, reports showed examples of 

— erroneous decisionmaking criteria 

— -programing errors, and 

— inadequate data controls. 

Ke discussed the concept of independent preimplementa- 
tion reviews with the Deputy Director of the Air Force 
Office of Data Automation. He agreed with the concept of 
such independent reviews but preferred that the reviews be 
made by independent teams within the design activity. He 
believes that auditors should become involved in evaluating 
designed or modified applications as soon as possible after 
the applications are placed into operation. 

Despite not making a savings .analysi s or< preimplementation 
changes, the Air Force Audit Agency bel ieves ■ that ■preimplemen- 
tation reviews should continue because: 
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— The quality of data systems is improved as a result 
of Air Force Audit Agency reviews. 

— The dollar impact of resources managed bv many auto- 
mated systems is a orooer subject 'for special audit. 

--Systems audits during the development stage help in- . 
crease the auditor's knowledge of. the systems. 

— The ability to make effective and efficient follow-on 
audits of operations is enhanced by the pre implemen- 
tation reviews. . ; 

Testing • 

After the designed or modified amplication, program is 
coded, agencies test the logic to determine whether the pro- 
gram will run ana will perform the Drocessing desired by the 
user. A description of" the nature of testing by each agency 
follows. 

— Pregr amer s at the :Javv FMSO preoare predetermined 
test cases and files to test the logic of the pro- 
gram. If tne results are satisfactory, the user op- 
erates the program with a duplicate ADP file and a 
selected number of actual transactions, which varies 
with each application. . Some of the selected transac- 
tions are traced to determine if the program is op- 
erating as intended and whether the decisions being 
made are the same' as ooerating Personnel would make, 
under the circumstances. The user advises FXSQ if 
there is a .problem. ' 

— Prog raters and designers, at SSA test both test cases 
and actual transactions. The number of selected 
transactions will vary, depending on the complexity of 
the program. The user is required to certify that 
the program is operating according to the user 1 s ' re- 
quirements. 

— VA primary testing is done by independent system 
auditors assicnea to the Department of- Data Manage- 
ment. The system auditors are independent of the 
prog r avers and designers* although they also work 
for tne sane department . The system auditors use a 
large number of' test cases that have been developed 
and reused during the years. An automated comparison 
of tne Drocessing is . made before and after the logic 
changes, ar.a the aif ic-.rences’ are printed out. Unless 
tb=re are ns nv differences.', alu.are reviewed for cor- 
rectness. Tne cases tnat are not printed are not 
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reviewed. The auditors' must certify that the logic 
conforms to t lie user requirements or issue exception 
reports when it does not. 


Federal officials recognize that '."the current state of 
the art. in program test i ng is imperfect. . According to Offi- 
cials of the Institute for Computer Sciences and -Technology 
Of . the National Bureau of Standards', most test procedures 
currently used do not insure that all lines of codes have 
been exercised. Officials at the agencies visited agree 
that it is virtually impossible to test for every condition, 
but say they do the best they can by ; 

- — testing as many conditions, as considered feasible and 
necessary and 


— adding to test case material conditions which caused 
problems during operations out had not been identi- 
fied during the original test, phase. 

The Institute and the Air Force consider the : test phase an 
area where the current state of the art must be advanced. 

The Institute was aware of numerous examples of com- 
puter .application programs which were considered to be ade- 
quately tested but which, during operation, rem into serious 
problems and caused incorrect actions-. As a result, -the In- 
stitute. in cooperation with the National Science -Foundation 
worked on methods to improve the state of the art.. 

One recently developed procedure is a software program 
that will monitor. tests of computer application programs 
written in FORTRAN (a programing language) . This program 
counts the number of times each line of code has beer 
exercised by test cases. Even though there is no insurance 
that every conceivable condit ion 'will be tested, there is 
insurance that each line of code has been tested at least 
once. Until recently, this capability was not generally 
available. - 

In a February 19/2 report , the Air Force said that 
so ft ware design and testing were the two most critical prob- 
lems in ADP requiring further research and development. In 
July 1973 the Air Force entered into a contract for the 
development of the type of software device that the Insti- 
tute had developed but. for a different programing language. 

Mon i t or i no o f_p i ccr am op e r at ion 

VA and the Navy largely rely on (1) internal auditor's 
reviews and (2) feedback from people affected oy bad 
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decisions or operating personnel to identify automated 
decisionmaking application problems. No form.al systematic 
monitoring of the applications' output is made, with one-ex- 
ception; VA audits education payments to veterans in excess 
of" a predetermined . amount . We believe that this is of lim- 
ited value in identifying, many costly systematic problems in 
automated decisionmaking applications because some types of 
transactions will never be reviewed. 

SSA has a formal monitoring group continuously taking 
random sambles of automated decisionmaking application out- 
put. According to SSA officials, this sampling has identi- 
fied design and programing errors and repetitive data errors 
causing erroneous payments in operating automated decision- 
making applications. Examples of t r«e kind of errors identi- 
fied by this monitoring function include: 

— Design, coding, or data problems in the automatic 
computation or recomputation of initial or ■ subsequent 
benefits. 

— Data problems in processing notices which affect pay- 
ments. 

— Design or coding problems in the updating of master 
data records (ADP files). 

——Inadequate preparation of data. 

SSA ' told us that system design and coding errors, at 
well as systematic repetitive data errors, were corrected as 
a Lesult of this procedure. However, it could not give us 
statistics on numbers of errors found or t'neii potential 
monetary impact, because SSA did not have this kind of 
information. !_/ 

SSA requires categorizing, in audition to monitor inn. 
the reasons for required program modifications. Tne cate- 
gories include: 

— Incomplete or incorrect performance requirements or 
program specifications. 

--Logic errors or program omissions. 


1 /Monitoring procedures are not always carried out. as soon 
~ as new Programs are placed into operation. The supplemen- 
tal security income’ program, an automated decisionmaking 
application, did not have full-scale monitoring during its 
initial operational periods. 
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— Incomplete validation of input data. 

— System-produced data not. in accordance with specifi- 
cations. 

— Incomplete testing. 

HE5f headquarters said that, a consulting firm noted a 
need for continuing reviews and evaluations of, among other 
t.hinqs, applications software. The firm suggested that a 
four-member team, including an auditor, be responsible for 
reviewing selected applications on a short-term cyclic basis 
including (1) reviewing t he ' appl icat ion against the original 
specification to determine that the software, was performing 
as intended and (2) determining whether application programs 
had been adequately modified when the processing circum- 
stances changed. HEW did not accept, the firm's report 

Data control . 

TIk sources of data input vary for the following loca- 
tions. 

— Navy ASO, Philadelphia, receives much of its data 
from external sources including (1) contractors for 
new aeronautical equipment, entering the supply system 
and (2) other Navy activities that receive , store , 
and issue aeronautical equipment. 

— -SSA, Baltimore, Maryland, receives most of its. data 
from about 1,300 offices and centers throughout the 
United States. • 

— The VA data processing centers in Hines, Illinois, 
and Philadelphia, receive data from several VA sta- 
tions throughout the country. 

Internal controls 

Our review shows that , even though written procedures 
may not exist., agencies develop and program extensive edit 
checks in software to help insure the validity of data coming 
into the system. Agency officials' admit 'that , . although ex- 
tensive work is done to analyze potential data errors during 
the design process, edit checks cannot be designed to iden- 
tify all types of data errors. 

In many cases erroneous, but acceptable data may be 
placed on input documents. Because such data, can represent 
a valid situation, there may be no way to design an edit, 
check to insure that it is correct. Also, edit checks will 
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not catch errors not . conceived -of — and therefore not 
considered— in designing edits. ... 

Agency officials agreed that, because of the detail and 
complexity involved in the design process, potential edit 
checks may be missed. . 

Examples of the types of checks observed at the three 
agencies visited included ■ 

— edit checks for incomplete, data, elements; 

—reasonableness checks; for example, rejected documents 
conta in ing numer ical values above or below a predeter- 
mined amount in a given data element? 

— logical checks; for example, checks for impossible 
conditions, such as negative inventory balances or 
alphabetic characters contained in data elements that 
were designed to contain only numeric characters; and 

--data relationship checks? for example* comparing data 
elements with other data on the same input document 
and/or contained in ADP files. 

External controls . 

Because agencies receive input from numerous sources, 
we limited our study of external controls to the controls 
at the agencies actually visited {VA Center ASO , and SSA). 

VA has written procedures for several external con- 
trol functions which include (1) random sampling of 
input documents to identify, and develop statistics 
which are used for identifying error rates and error 
sources, (2) selected verification of eligibility 
data contained in A BP files, (3) date stamping and 
sampling of documents to control the timeliness of 
documents processed, and (4) controls over unproc- 
essed (pending) documents. 

— ASO makes no manual reviews of supply-related data 
received from Navy activities and' therefore primarily 
relies on { l ) controls at the data "preparing site and 
{2} internal controls designed in the ASO software. 

ASO makes selected manual reviews of data received 
from contractors on new aeronautical components before 
the data is allowed to enter the system. 
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* — SSA basically relies or. the (1) internal controls 
. designed into the software, (2) end-of-line monitoring 
procedures;, and (3) manual reviews at the vast num- 
• bers of offices and centers preparing the data. 
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CHA PTER 6 

' AUTO HAT ED D EC IS 1 ON MAKING APPL I C AT IONS CONT I NU E __TO 

MAKE BAD-D ECISIONS UNTIL PROBLE M S. ARE CORRECTED ‘ 

Errors made by users, designers-,' and programets of 
automated decisionmaking applications, if not identified 
and corrected in the-, review and testing phases of the design 
process, can cause bad decisions which will continue until 
the errors are detected ana corrected. When an insignifi- 
cant error for a given action is multiplied by thousands or 
millions of the same type of actions over a period of time, 
the .’ei for is compounded. Unnecessary costs -w'il 1 grow and 
become large. An error allowed to exist. for 5 years will 
cost the Government more than if the er ror , is detected and 
corrected within, for example, 3 months after the automated 
decisionmaking application is in operation. 

ERROR DETECTION 

In previous chapters we discussed what agencies do to 
detect design and data problems. , Because errors get through 
design and test processes and because data errors are made, 
early detection of them is important- in reducing the cumula- 
tive effects of bad decisions. 

ERROR CORRECTION 

Detecting errors occurring in automated’ decisionmaking 
application, software and/or data will not, by itself, stop 
the unnecessary costs being incurred. When detected, action 
must be taken to correct the errors by modifying the soft- 
ware, or improving the data quality, or both. 

We have noted some instances in which problems were, 
identified but corrective actions were not taken for a long 
time. An example follows. 

Kav y use. of overstated dema n ds in . * ■ 

automated deci s i cnmak l n q ap pi l ca t i ons * 

A GAD report, B-162152, May 21, 1974, noted that in 
Havy auditors saw a need to design a routine in the standar 
computerized supply management system used by Navy inventor 
control points for removing from ADP .files past material 
usage quantities (demands) associated with ' canceled requisi- 
tions. Ine demands recorded .in these ADP files were used by 
several ' automated decisionmaking -applications; 
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The report noted that in .3*69. Saw consaand officials 
agreed with the need to properly adjust demand forecasts 
for invalid oroers but said that it. would not be able to 
.correct the problems before 1271 because of other orioritv 

T n r 1 9 7 2 T ° ^ h p S ^ ° f fc Said th ^' at the the . GAO review 

in 1972 , L he Navy was stilj. not eliminating from ad>> files 
-demands related to invalid orders-.. .- 

We estimated that about $34 million in invalid demand® 
ADV fil6S and th ^' these overstated demands ' 
ted m unnecessary materiel buys and reoairs totaling 
about $10 million a year. At least $3. millibn in annual 9 
unnecessary costs were initiated by automated decisionmaking 
applications using this overstated- demand data.' 9 


, The design change to correct the condition 
made at the time of this review, so we discussed 
tor the delay with appropriate Navy officials. 


had not been 
the reasons 


We were told that, 
tion received from the 
project was established 
design modification. 


because of the GAO report and direc- 
Depar .ment of Defense, a high—or ior ity 
on Jane 14, 1974, to make the' needed 


The reasons cited by Navy officials ‘‘or the 5-v«ar d«la 
in initiating the modification included ■ ~ ' ' 

disagreements within the Navy oh whether all canceled 
requisitions should result in reducing recorded de- 
mangs t 


“high-priority workload 
by higher headquarters 
Department of Defense, 


at the design activity mandated 
levels in Doth the Navv and the 
and 


— lack of pressure olaced. on the Navy command and desiqn 
activity by the inventory control points since reduced 
demands could result in budget reductions. 


AGENCY PROCEDURES FOR TIMELY CORRECTION 
_SOFTWARE_DESiG;4 PROBLEMS - ' " ~ ' 


Agencies establish priorities and target dates for soft- 
ware assign and modification projects. Agency guidelines 

aJd°co-n^f e cost-benefit studies to justify establishing 
and committing resources to a large design effort. 


According to 
attention is give 


some Feceral officials, 
to doing cost-benefit 


however , little 
studies which 


45 


Approved For Release 2002/06/05 : CIA-RDP79-00498A0003001 10009-7 



Approved For Release 2002/06/05 : Cl ArRpP79-00498A0003001 10009-7 


demonstrate either (1)' how much will be saved by eliminating 
an automated decisionmaking application problem that exists 
or (2) how much the continuing automatic decisions will cost 
the Government if the problem is allowed to go unchanged. 




Approved For Release 2002/06/05 ; CIA-RDP79-00498A0003001 10009-7 


CHAPTER 7 

OPINIO NS ON WAY S TO PREVEN T OR REDUCE 

, t he impact op problems i n 

AUTOMATED DECISIONMAKING APPL ICATIONS 

We believe that, despite the imperfect state of the 
art in application design and the widespread problems of 
getting quality data to the computer#' every Federal agency 
asi rcg these appl icat ions should consider doing certain 
things to prevent or reduce the impact of the problems 
identified in this report. ~ ' 

We issued a questionnaire to 200 members of each of the 
following professional ^associations that are dedicated to 
furthering the quality of ADP-produced products: 

x — The Association for Computing Machinery ' s Special 
Interest Group for Business Data Processing! 

: — The Association for Computing Machinery's Special 
Interest Group for Management of Data. 

"The Society for Management Information Systems. 

The quest ionnaire described the various problems that 
ve had observed in both the software design and data areas 
and requested the members to rate possible .solutions pre- 
sented in terms of their effectiveness and cost benefit. 

The ratings were designed to determine the validity of each 
solution, assuming each application involved spending millions 
of dollars or had an impact on people. 

Some of the solutions can be. applied before the applica- 
tion becomes operational to prevent problem conditions. 

Some of the solutions were .to be applied after the automated 
decisionmaking application became operational to detect prob- 
lem conditions early. If timely correction is made, the im- 
pact will be reduced. 

A total of 253 people responded to the questionnaire. 
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Summary of People 
Answering the GAO Ques tio nnaire . 

Portion of questionnaire 


Affiliation of data 
processing professional 

Design 
and data 

Design 

only 

Data 

only 

Total 

Commercial concern 

136 

45 

4 


185 

Academic 

34 

10 

1 


45 

Government 

25 

1 

i 


27 

Hot indicated 

4 

_2 

_0 


6 

Total 

3J)9 

58 

_6 


263 

POSSIBLE SOLUTIONS — SOFTWARE PROBLEMS 





Some of the highly 

rated solutions to the 

various 

de- 


sign conditions are: 


--Documentation should be prepared that highlights (1) 
key portions of the automated decisionmaking criteria, 
(2) data elements that are critical to the decision- 
making, and (3) the edit checks placed (or justifica- 
tions for omitting them) in the software. A formal- 
ized synopsis of these items should be prepared for 
review and approval by top management. 

—Qualified auditors or others who are independent of 
designers and users should review the designed appli- 
cation before it is placed into operation. Others 
could include a design team independent of the origi- 
nal designer and user. They would be responsible for 
evaluating the (1) adequacy of the decisionmaking 
cri*»ria, (2) logic in the coded application/ and (3) 
nee and uses of edit checks to detect incomplete 
dat. • elements put into the application. 


— Similar independent teams should review the operation 
of these applications shortly after they are imple- 
mented. Trie objectives would be to evaluate the^ade- 
quacy of the deci sionmaki ng criteria in an operational 
environment and to provide for early detection of any 
bad decisions. This would allow for early correction 
of problems. 

— Some form of cyclical system monitoring of actions 
initiated by operational automated decisionmaking ap- 
plications should exist. Teams composed of (but not 
restricted to) designers,- users, and auditors could 
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analyze application-initiated actions to ( 1 ) see if 
desired results were achieved the best wav, (2) 
identify unforeseen circumstances that would require 
■edifying the application, (3) determine that the 
■ actions were as the user and designer in ended, and 
*12 tf ? at decisionmaking was not adversely 

^i? C checkf 1RCOB * Plete data >t being screened by an . 

“the same g n?^^ USer f ould ^ physically located in 
the same place during design phases to allow for con- 

J ff ni l atl 5 n * In effect ' the design would be 

,J.'f effort and would help to insure that adequate 
decisionmaking criteria were contained in the applica- 

?° uld be established for software modi- 
fication (changes) which are at least oartially based 
on the cost or continuing incorrect automatic actions 
if no changes are made within a short time* 

T! e inifcia tor of the needed software modification 
i"hL!f¥ e I , headquarters, user, audit team, and/or 
o heto) should be informed about the status of the 
cnange and be provided with confirmation that the 
changes have been made. • . 

P OSSIBLE SOLUTIONS — DATA PROBLEMS 

=o„d i u:L 0 a £ « h fo? i9hly •*»“ S01 V U :°" S to th* various data. 

—Establish followup procedures for insuring the ( 1 ) 
imely receipt of data preparation instructions and 
(2) use of instructions by data preparers. 

t n Uaining the importance -of complete and 
correct data on computer input documents. 

Make selective manual verification of key data on 

documents and in A DP files with hard copy docu- 
me,.ts ana with the data originator. 

organization (data base adminis- 
, thrft could as responsible for the above steos 

f alua tmg and testing internal and external 
fnd used 13 er ‘ pl ° yed 2nd inpu t documents designed 
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CH APTE R 6 

CONCLUSIONS , R ECOMMENDATIONS, M D 


AGENCY COMMENTS ' , • 

Automated decisionmaking applications initiate the 
spending of billions of dollars a year without anyone re- 
viewing and evaluatinq the individual actions. They are 
also used to support a multitude .of . functions that, although 
not directly related to money expenditures, can affect mis- 
sion achievement and make, dec i si oris regarding individuals. 

Many of these applications make bad decisions because 
of various software and data problems." The causes, of tie 
problems are numerous. 3ad decisions may result m unneces- 
sary costs and overpayments of hundreds of millions of dol- 
lars a year— exactly how much is unknown. Such bad deci- 
sions can also impair mission performance and harm individ 
u a 1 s . 

In the current imperfect environment , the cnances of 
continuing bad decisions and unnecessary costs are great. 
Actions are needed. We believe that it is necessat ; ne ^ e 
fore to develop and issue Federal-wide guidelines t« oster 
uniform cost-effective practices that will (1) minimize the 
chances of problems occurring, (2) detect as soon as. pos- 
sible the. problems that do occur in operating automateo de- 
cisionmaking applications, (3) correct problems as early as 
possible to reduce their adverse impact, and (4) insure 
that toe practices are being ' ef fectively applied. 

Some practices we consider necessary to meet these ob- 
jectives already exist at some agencies. For instance, we 
observed testing, joint design, and inclusion of internal 
data controls. We also observed ' some established data man 
ageaent practices which could identify data input problems. 

Several practices considered by us and. by data process- 
ing professionals to be cost effective in reducing the 
chances or impact of bad decisions were not being applied to 
all crucial automated decisionmaking applications. This ind 
cates a need for central guidelines in such areas as: 


— Preparing document at ion and/or a formalized synopsis 
that highlights, for example, key decisionmaking cn 
teria, data elements critical to the decisionmaking, 
and edit check placement to facilitate thorough re- 
views by others. 
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— Making. .pre implementation reviews of the desian^d or 
modified, applications and. interna] ' data controls. 

The reviews should be made by .groups that are in- 
. ^Pendent of the designer or user. The groups should 
consider evaluating., amonq other ' things , the ( .1 ) ade- 
?K 3Cy °V' he d ? cis ionmaking criteria, (2) logic in 
. the coded application, and (3) needs and uses of edit 
checks contained in these applications* 

—Analyzing actions initiated by these applications as 
soon as possible after they are placed into operation 
co insure that (1; they are operating as intended, 
cl intended operation is the most economical and 
effect 1 ve method , and (3) c.i r cumstances that were not 
con si dei ed duiin*} design have not at i sen# 

—Cyclical or ongoing monitoring of automated decision- 
making application output to insure that (1) desired 
results are achieved most economically and effec- 
tively, (2) new circumstances, have not arisen, that 
will require changes to the decisionmaking or other 
processing criteria, (3) the logic is correct, and 
(4) decisi.omaking is not adversely affected by incom- 
plete data not being caught by an internal edit check. 


-Establ isbing 
modification 


priorities 
which, are 


and target dates for software 
it least partially based on ti 


...wxi-^awon wni cn. are at least partially based on th. 
unnecessary costs of continuing incorrect automatic 
actions and keeping the initiator of -modifications in> 
tor meo of the status of the changes. 

establishing a single point in each organization that 
would have prime responsibility for insuring that 
these applications are making decisions based on the 
best data available by ( 1 ) eval uat i ng and testing the 
ciata and data controls (internal and external), (2) 
adequately, training data preparers, (3) reviewing th» 
adequacy and currency of instructions given data pre- 
parers and insuring they are complied with, and / 4) 
insuring that forms designed for data processing' 
minimize the chances of data errors. 

To begin focusing on west should be ' managed , lop man- 
agement in each agency should be.. aware of the automated de- 
■SvPl^nM applications that exist (operational and under 
other l ’ ~ functions they support, their monetary and 

■ . '-‘ ; ? act s, nature and sources of input, the ourout- 

. - n i 1 1 a e.. actions, the programed reasons for any manual in- 
tervention, and otner important, chat acted i st ics. 


ol 
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Aqencies should be required to take stock of their 

•automated decisionmaking apolicat ions. .This act ion shou 

include ascertaining whether their current oraotices for en- 
veloping, .modifying, and -operating- such application-,- to 
gether with related data controls, are adequate to surface 
problems of • t he types discussed. Guidelines .should be. 
issued to. . indicate cost-effective corrective procedures, 
and agency management should insure that automated deci- 
sionmaking applications are under control-. 

RECO MMENDATION S 

We believe that, si nee automat ea decisionmaking appli- 
cations have- not, previously been recogni zed as _a separat e 
problem area requiring management attention anr, since mil 
lions of dollars are presently being wasted as the result 
of actions generated by such' systems, the Office of Manage- 
ment and Budcet (OM3) should act immediately to improve tne 
situation. Specifically, we recommend that 0M3, ir. its over- 
sight capacity, require that: 

— -Each agency determine whether any of its computer op— 
stations involve automated decisionmaking applica- 
tions. . 

The agencies review each operation to determine ^ 

whether incorrect actions are being taken as. a result 
of these applications. (Pending -issuance of technical 
guidelines by the National Bureau of Standards for- 
making such reviews, the agencies should examine ^ 

- enough automatically generated decisions to provide a 
basis for deciding whether incorrect decisions are 
occurring and, if so, should take the necessary steps 
to correct the situation causing the incorrect deci- 
sions. 1 . 

— Before any new automated decisionmaking, applications 
ate initiated by an agency, the proper steps ate taken 
to insure correct, decisions. This would include, pend- 
ing issuance of the National Bureau of at andaido 
guidelines, a carefully, chosen .combination of ^inde- 
pendent review of systems design , adequate t eSt ing 
before implementation, and periodic testing o£ de- 
cisions after implementation, as discussed earlier 
in this repot t . 

—Agencies report on the action? taken and establish 
an appropriate mechanism for monitoring such reports. 

We ' recommend that, because the Nat ional • Bureau of Stand- 
ards has responsioi 1 i t i es for. technical aspects by ASP, tne 
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' Secretary of Cotntis-tcfc- direct the Bureau to - issue technical 
guidelines for developing, using, technically evaluating, 
documenting, and .modifying these applications in the Fed- 
eral Government. When issued, these guidelines should con- 
tain certain criteria for independent technical reviews and 
ror ifconitoring of these applications to insure problems are 
detected and corrected promptly. The General Services Ad- 
ministration should incorporate Bureau guidelines in its 
agency directives. • " . 

In addition, we recommend that: 

--As GSA suggested, the Civil Service Commission de- 
velop and add to its ADP. training cut.ticulun courses 
in automated decisionmaking a poll cat ions so that 
managers, technical personnel, and auditors will be- 
come better eguippeo to deal with them in an aporo- 
priate manner. 

Internal audit groups in agencies having automated 
decisionmaking aocl icat ions par t icioate act ivelv in 
design, test , and reviews of such systems to carry 
out their responsibilities. 

finally, ws suggest that the . Joint Financial Management 
Improvement Program consider this area for ongoing, attention 

are . sending copies of this report to all departments 
and independent agencies for their information, use, and 
guidance pending issuance of the 0MB and National Bureau of 
Standards material. 

AGENCY COMMENTS 


We issued the proposed report to several agencies for 
comment. Their replies indicate general agreement as to the 
problems reported and vatyinq ooihions on the recommsnda- 
t ions. 


With respect to the problems, the Associate Deout y Ad- 
ministrator , VA, agreed that there was a need for sound man- 
agement of current large sophisticated data orocessing sys- 
tems. He said the report was useful, in' identifying' and con- 
solidating the problems associated with automated decision- 
making applications. He believes that the formulation of 
standards relating to these applications is imperative. 


The Assistant Secretary, Comptroller, HEW, said that 
no one would disagiee that software and data problems exist 
ana that such problems could result in automated decision- 
marling applications ; hat made erroneous decisions in some 
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cases. He believed that as rmich emphasis should be placed 
in preventing software errors as in detecting and correcting 
them. He agreed that the current state of the art. in soft- 
ware development, could not assure error-free software. 

The Assistant Secretary of Defense, Comptroller, said 
that, most of DOD's automated systems fit the definition of 
automated decisionmaking applications, although damage re- 
sulting from errors in some systems was less direct and less 
measurable than in disbursing and supply systems. He added 
that our statements of possible solutions to software and 
d$ta problems are logical and constructive and that while 
they are similar to many DOD practices, their documentation 
will assist system developers, auditors, and operators. 

The Acting Administrator, GSA, sait- that the report 
performed a valuable service in identifying automated de- 
cisionmaking applications as an area of data processing con- 
cern and, as such, warrants wide circulation to ADP software 
managers in the Federal Government. He strongly agrees with 
our solutions for software and data problems, including 

— preimplementation and post implement at ion system re- 
views by independent groups and 

— cycl i cal system monitoring. 

The agencies had varying opinions on the tentative 
proposals contained in our proposed report. We- have 
weighed their comments and considered them in formulating 
the proposals in this report. For example, we prooosed 
that the agencies involved report to GSA on actions taken 
in response to our recommendations. Upon consideration 
of the responses to our proposed report, we have modified 
our recommendation to provide for OMB to determine ari 
appropriate reporting mechanism. 

Also in response to our proposed report, the Acting Ad- 
ministrator, GSA, suggested that the National Bureau of. 
Standards could develop Government -wide guidelines for i n “ 
formation systems development which could specifically in- 
clude automated decisionmaking. 

On January 12, 1S76, we discussed the suggest ion. wit h 
the Director, Institute for Computer Sciences and Technology 
National Bureau of Standards, who agreed to the need for 
Government-wide technical guidelines that would include 
developing, using, modifying, reviewing, and monitoring auto 
mated decisionmaking applications and said that budgetary 
resources would be solicited for the National Bureau of 
Standards to perform this task. The guidelines, when 
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» 

completed, would be issued as part of the Federal information 
processing standards series for use by Federal agencies. 

We informally discussed the recommendations with 0MB 
officials who have responsibilities in the ADP area. They 
believe that, the report points out important problems in 
this area and agree that issuing policy guidance is appro- 
priate. • • . 

We discussed our recommendations to the Civil Service 
Commission with officials of the ADP Management Training 
Center who agreed to further emphasize controls in their 
ADP training. ' 
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DEPARTMENT OF HEALTH. EDUCATION. AND WELFARE 

OFF IC£ OF THE SECRETARY ■ 

WASHINGTON. D C- 10201 


m'l.l 1975 



Hr. Gregory J. Ahart 
Director, Manpower and 
Welfare Division 
U.S. General Accounting Office 
Washington, D.C. 20548 

Dear Mr. Ahart: 

The Secretary asked that I respond to your request for 
our comments on your draft report to the Congress entitled , 
"Improvements Needed in 'Managing Computer-Based Automated 
Decisionmaking Applications in the Federal Government. 

They are enclosed. 

We appreciate the opportunity to comment on this draft 
report before its publication. 

Sincerely ^ours. 


John D . Young 

' — ■"Assistant Secretary, Comptroller 

Enclosure 
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APPENDIX I 


COMMENTS ON GAO'S DRAFT REPORT ENTITLED 
"IMPROVEMENT NEEDED IN MANAGING 
COMPUTER-BASED AUTOMATED DECISION MAKING 
APPLICATIONS IN THE FEDERAL GOVERNMENT" 


APPENDIX I 


OVERVIEW 

The draft report identifies a certain type of EDP application 
vmich GAO calls an Automated Decision Making Application, 

CADMA) , and notes that ADMAs are widely used by Federal 
agencies. The report points out that the distinguishing 
characteristic of ADMAs, as compared to other computer 
application programs, is that many of the actions initiated 
by tne computer take place without review and evaluation 
by people. According to GAO, there are indications that 
funds are being wasted because of incorrect, unreviewed 
actions. 

The report discusses, at some length, the use of ADMAs by 
federal agencies, and points out that ADMAs can make bad 
decisions. It categorizes the causes of these bad decisions 
~s being software problems or data problems, then goes on 
to identify and discuss the reasons for these problems. 

Mo one will disagree that software and data problems do 
exist, and that such problems can result in ADMAs that make 

de . clSAons in some cases. It is of utmost importance, 
therefore, at such problems be prevented during the desiqn 
and implementation of the system. While we are of the opinion 
that trie current state of the art in software development 
techniques and test techniques cannot assure that, error free 
software can be designed, techniques are available that can 
contribute significantly to the reduction of software errors 
Furthermore, practice has suggested that the method- of 
organization of a development effort can have a favorable 
■ L ** iact on . thc Grror level as well as the development cost. 

™«r the Statfr ° f ih c c art of development and testing techniques 
cannot assure error-free software, it is of equal importance 
affo reviews Oi systems take place before operation, shortly 
after implementation, ami on continuing or cyclical basis 
, ?P e , ratlona1 ' sysfomu . The extent of review of an ' A DMA 
should be a function of the probability and impact of errors. 

The report discusses various ways to prevent or reduce the 
impact of problem conditions in ADMAs. In our opinion, the 
possible Solutions mentioned in the report are, for the ^ost 

Wo " ld , howcVl ' r - P Jac -' a greater emphasis 
th«-n undo in the (.Ao report on (I) involvement of the 'user 
an the development oL ,.n A DMA and. .(2) appi caches to reducing 
probability of error: .it the design and Li st stages rather 
nan emphasizing error u.-u-ctioii and correction in the 
operational stage. . 
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GAO concludes that the development and use of ADMAs is 
necessary but because of the current imperfect en ircrunent, 
chances of continuing bad decisions and unnecessary costs 
are great. Consequently, GAO believes that it is necessary 
to develop Federal-wide policy to foster uniform cost- 
effective practices that will minimize the chances of 
problems occurring, detect the problems as early as possible, 
and assure that the practices are being effectively applied. 

G AO RECOMMENDATIONS AND HEW COMMENTS 

RECOMMENDATIONS 

Because GSJ is responsible for developing Government-wide 
policy on ADP management and for seeing that the policy is 
carried out by the departments and agencies, GAO recommends 
that the Administrator, GSA: 

— Require the identification and characterization of 
ADMAs used- by Federal agencies. (A starting point 
for material to be included can be the types of data 
GAO obtained during its study of ADMAs — volume of 
transactions, impact of decisions, etc.). Thi^ ill 
provide agency management and auditors with bas 
information on where their resources could best be 
applied. 

-- Issue policy requirements and guidelines for the 

.management of ADMAs in the Federal government. Most 
importantly, the policy rind guidelines should 
establish criteria for independent reviews and 
monitoring of ADMAs to assure that problems are 
detected and corrected in a timely manner. The policy 
should also include criteria for cost-effective 
devclooment , modification, documentation, review 
and testing of ADMAs. . . 

Require agency reporting, concerning (-1) actions taken 
based on the criteria and '(2) problems identified and 
corrected as a result of independent reviews and. 
monitoring of ADMAs. Justification of cost effective 
ways of managing ADMAs should be included. 

HEW ’ COMMENTS • 

With respect to GAO's first rc-cnmmendat ion , . ve do not believe 
that it would be useful to have all agencies identify and 
characterize their ADMAs. To. do so would result in the 
preparation of an enormous volume of reports covering 
hundreds of ADMAs. Since xt is unlikely that GSA would be 


53 


Approved For Release 2002/06/05 : CIA-RDP79-00498A0003001 10009-7 



Approved For Release 2002/06/05 : CIA-RDP79-00498A0Q03001 10009-7 


APPENDIX I 


APPENDIX I 


able to effectively utilize these reports, their development 
and preparation would be a waste of agency time^and resources. 
For '.similar reasons we do not favor. .GAO* a third recommenda- 
tion which would require agencies to submit reports concerning 
the actions taken pursuant to GSA policy directives. 

We agree in principle with the second recommendation -- that 
GSA establish guidelines for the management of ADMAs in ..he 
Federal Government. The establishment of guidelines woula 
encourage agencies to utilize acceptable practices for 
developing, modifying, reviewing and monitoring their AD A 
systems, • 

We are of the opinion that such guidelines as GSA might 
develop must be flexible cu recognize that ADMA systems are 
of varying complexity and of varying impact in terms o. 
probability and cost o£ errors. Thus, practices employed 
for the development, modification, review, and monitoring 
of a particular ADHA should be oriented, towards overall cost 
reduction, i.e., expected cost of errors plus c °st 
development, modification,... In light of the diversity of 
ADMAs, we do not believe that it is practical t^ establish 
■policy requirements* at this time. We believe that a more 
effective procedure would be for GSA to issue guidelines and 
then to periodically conduct on-site reviews and audits of 
various agency ADMAs. The objective of such review jrfould 
be twofold:- (1) determination of the extent to which guide- 
lines were being followed by agencies and (2 ) determination 
of the effectiveness and efficiency of the recommended 
practices so that they could be developed and refined based 
on actual experience. 


Furthermore, as we indicate in the Overview to those consents, 
we believe that efforts to eliminate errors during develop- 
ment is of equal importance to the review and monitoring 
efforts. Therefore, we suggest modifying the second recommen- 
dation to read: 

"Issue guidelines for the management of ADMAs in the 
Federal Government . These guidelines should include, 
recommended practices and criteria for cost effective: 

I. development, modification ana testing of ADMAs 
to reduce error levels in software and data 
collection. ^ 


2. documentation of ADMAs for internal and external 
uses, 

3. review and ronitorino of ADMAs both as continuing 
activities by systems and user personnel and by 
independent groups." 
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OTHER 'COMMENTS AND SUGGESTIONS ON THE REPORT 

1. In the third paragraph on page 5« of the draft report, 
a statement is made that SSA official said that they 

assume that designers and programmers are adequately 
trained and experienced and that such instructions are 
not necessary, “ This is not an accurate statement . 

. We suggest that GAO change the sentence, to read: 

"An SSA staff member said that designers and 
programmers are adequately trained and experienced 
since there are continuing courses offered in 
systems design so that skills can be maintained ac a 
satisfactory level.” 

,2. In the last paragraph on page 65, the second sentence 
reads ^According to SSA officials, this sampling has 
identifi d many design and programming errors and 
repetitive data errors causing erroheous payments in 
operating ADMAs . ” The word "many" is misleading in 
that this is an end of the line operation and most 
errors are discovered in validations, etc. long before 
these operations are performed. The sentence should 
read: 

"According to SSA staff members, this sampling has 
identified design and programming errors and repe- 
titive data errors causing erroneous payments, in 
operating ADMAs . 11 

3. The first paragraph on page 66 begins. "SSA advised us 
that many system design and coding errors, as well as 
systematic repetitive data errors, are corrected as a. 
result of this procedure." For the same reasons given 
in the preceding paragraph of our comments, the word 
"many” is misleading and should be deleted. 

|See GAO • note , p- 61.] 



There is 
chapters 
Chapters 


considerable overlap and duplication in several 
of. the report- In particular, we suoqest that 
3 and 4 be combined to improve readability. 
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6. We believe, in general, that the report tends to under- 
play the importance of the user in the development of an 
ADMA. We note with interest that in the opinion of 
“people answering the questionnaire" (page 36 of the 
report) the most often cited problem is "inadequate 
conatunications Detween the parties to software design." 
The second ranked problem in this list is "incorrect 
perceptions of the nature of the actual transactions to 
be processed." 


Furthermore, in Chapter 7, "Opinions on ways to prevent 
or reduce the impact of problem conditions in ADMAs," 
respondents from several professional organizations 
suggest that "Physical, collocation of the designer and 
user should be accomplished during the design phases 
to facilitate constant communication. In effect, the 
design would be a joint effort and would help to insure 
adequate decision-.?^ Ding criteria contained in the ADMA. " 
Despite the importance of these caupes of errors and of 
this recommendation of professionals to overcome them, 
the policies advocated by GAO in Chapter 8 fail to address 
the necessity of user involvement. 


Therefore, we suggest that the GAO report place greater 
emphasis on the participation and responsibility of the 
user in an ADMA system. In commenting on the draft GAO 
report "GAO Guidelines for Management Information 
Processing Systems," May 1974, HEW stated: "The 
Guidelines include the user in the system development 
from the standpoint of user education as opposed to user 
participation. While user education is important, it 
is not enough. The success or failure of a system is 
critically dependent on user involvement and participation . " 
We believe that this dependency is even more critical in 
an. ADMA syste;,. 

7- The importance of personnel selection and training for 

ADMA development, operation, monitoring and review should 
be given greater emphasis in the GAO report. Designers 
and programmers should be familiar with design tools and 
techniques, e.g., structured and modular flowcharting 
^Dd programming, decision tables, data base design tools, 
data element management, data collection alternatives. 
Management should be aware of alternative organizations 
for system development, e.g., chief programmer teams. 
Designers should also be aware of techniques for testina 
and monitoring systems including statistical samp. ing 
approaches. Knowledge can be obtained via government 
or private sector training courses. 


CjAO note: Material no longer related to report has 
been deleted. 
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VETERANS ADMINISTRATION 

Office or t«c Administrator or Vct£k**5 Affairs 
Washington., D.C. 2042v 

NOVEMBER 2 8 1975 . 

kr. Gregory J. A!>art 
Director 

Manpower and Welfare Division 
Ui. General Accounting Office 
Washington, D.C. 20548 

Dear Mr. Abart: 

We appreciate the opportunity to review / • d consent on your 
draft report relating to the management of automated decision-making 
applications and are in agreement that there is a need for sound manage 
ment of the large, sophisticated data processing systems in existence 
today. 


APPENDIX, II 



[See GAO note . ] 


Your report has proved useful in identifying end consolidating,- 
in one place, many the problems associated with automated decision- 
raakinz application in a clear, straightforward language. «e bel eve 
that the formulation of standards relating to these applications is 
imperative, and have already begun to draft our own general requirements 
and guidelines. 

Sincerely y 


‘ Assotufc AdR«8^t/alar-ia tfe sj 

RICHAKD L. ROUDEBCSH 
Administrator 



GAO note: Deleted comments refer to material discussed 
in cur draft report but. not included in this . 
final report. 
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UNITED STATES OF AVEB'.CA 
GENERAL SERVICES ADMINISTRATION 

WASHINGTON. DC 


DEC 29 1975 

Honorable Elmer B. Staats 
Comptroller General of the United States , 
General Accounting Office 
Washington, D. C. 20548. 

Dear Mr. Scaats: 


!. /^i 

Xrrp 7 


We appreciate the opportunity' to review your draft report ’’Improvements 
Needed in Managing Computer-Based Automated Decisionmaking Applica- 
tions in the Federal Government* '* 

The report performs a valuable service in identifying automated decision- 
making applications (ADMAs) as a discrete area of data processing 
concern and, as such, warrants wide circulation to ADP Software 
managers in the Federal Government. 

We strongly agree with the following GAO recommended spiv. l .io ns for- ' 
software and data problems: 

. Pre- implementation and post- in^lementation system audits 
by independent groups. . 

* Cyclical system monitoring. 

. Joint system design by users and A DP systems analysts. 

In addition to the management solutions mentioned in the report, there 
arc modern computer programming techniques which can aid in increasing 
the integrity of any system. Developing detail logic with decision tables 
rather than flow charts is particularly effective in data editing applications. 
The use of ' top down" programming and "chief programmer teams' is 
proving successful in minimizing errors. Employment of a data base ■ 
administrator throughout l>oth the developmental and operational stages, 
of a system will help assure that valid data is being processed. 


Keep Freedom in Tour Fuitse With US Savings &mds 
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While we generally agree with the conclusions and recommended solu- 
tions for software and data problems contained in the report, we do not 
agree with the recomn . 'ndations that GSA iss ue policy and guidelines 
for the management of ADM As nor that GSA require agency reporting to 
allow monitoring of agency performance. Rather wc would suggest, since 
A DMAs are prirt of the broader universe of information systems develop- 
ment, that: 

. The National Bureau of Standards, with GSA cooperation, 
develop government -wide guidelines relating to information 
systems development which chould specifically include auto - 
mated decisionmakir/g. 

. Agencies report to thrir own agency head regarding decision- 
making criteria, ADMA problem identification and corrective 
actions taken, and that these reports be made available for 
review by OMB and GSA, in line with review provisions in 
Federal Management Circular 7-1-5. 

. The Civil Service Commission include in its management 
training programs, a course on automated decisionmaking 
stressing the need for cost effective development, joint 
systems design by users and ADP systems analysts, systems 
monitoring and auditing of ADMAs. 

Because of the significance of this report, w had the opportunity to have 
the Ad Hoc Committee for Implementation of P. L. 89-306 briefed by a 
representative from GAO prior to issuance of this draft. This Committee 
is chaired by the Commissioner, ADTS, and representatives from ADP- 
intensive agencies are committee members. At a later meeting, our 
comments were discussed and the Committee generally concurred in the 
approach GSA is proposing. 

If you have any questions, please let us know. 
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Because of the unique- problems of automated systems , we have end will 
continue to develop and apply special measures to their quality control. 
However, they are in no way exempt frcm standard Federal accounting 
system certification, management reviews, and internal audit controls* 
The net effect then is to increase management control of automated 
systems in canp&rison to manual systems. 

Your statements of possible solutions to software and data problems 
are logical and constructive. While they are similar to. many DoD 
pi'actices, their documentation in a- compact set will assist our system 
developers, auditors and operators. 

With .'respect to the recorcaendations included in the draft report, we 
interpret ISA's charter in the ADP field to address procurement of 
ADP equipment, supplies and services* Your report .is aimed at a 
different arena, that of functional procedures and accounting controls. 
Accordingly, we recommend that: 

1. The subject be proposed as a matter of continuing interest 
by the Federal Financial Management Improvement Program. . The 
inter-agency effort of senior financial managers is an appropriate 
forum for exchange of new procedures and. techniques* 

2. Pertinent and documented studies, research reports-, methods 

and techniques be provided by developing agencies to the National 
Technical Information Service (NTIS) of the Department of Commerce 
for disseminati-cn at cost to other potential users in accordance 
with the ISIS charter. * 

3* The report be issued as a study, retaining the findings and 
conclusions but deleting the recommendations and substituting the 
following: 

"Each Federal Agency should review its internal regulations 
and procedures for management of ADMA systems to assure 
protection, of mission effectiveness and ' government reso'orces 
from system, errors. Each agency should establish, specific 
internal procedures to assure that internal controls and 
audit trails for error detection and correction are made 
a part of system design specifications, tested prior to 
system implementation, and included in routine and 
special audits throughout their operational life." 
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■* ' " 

Thank you for an informative and valuable research effort. The 
opportunity to comment on the draft report is appreciated. 

Sincerely 5 
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* INTERNAL AUDIT REPORTS ON 

AUTOMATED DECISIONMAKING APPLICATIONS 


• Title of 
report 

Army Audit Agency: 
Coordinated 
Audits of Depots 
(Maintenance 
Operations) 

U.S.. Army Train- 
ing Center, In- 
fantry and Fort 
Polk 

Direct Support 
System. 


Type of 
application 
Date involved 


3/ 4/74 Maintenance 
workload 
acceptance 

12/21/73 Requisition- 
ing 


10/16/73 Requisition 
processing 


P roble m- identified 
Software Data 

X X 

X 

X 


Materiel Obli- 2/ 8/74 Procurement X 

gation Valida- cancellation 

tion Procedures 


Cat a] og Function 8/21/73 


Naval Audit Set vice: 

Servicewide 12/ 6/73 

Audit ot the 
Aeronautical Re- 
parable Compo- 
nents Program 


Automated pro- X 
curement and 
requisition 
processing 

Overhaul sched- X 
uling 


X 


X 


Headquarters, 11/ 1/73 Requisitioning X 

Paci f ic . Missi le 
Range, Point 
Mugu, -Califor- 
nia 

Navy Aviation . 10/16/72 Redistribution X X 

Supply Office, 

Philadelphia , 

Pennsylvania 
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Title of 

report Date 

Naval Audit Service 
(continued) : 

Aviation Supply 6/15/73 

Office, Philadel- 
phia, Pennsyl- 
vania 

Navy Aviation 12/10/74 

Supply Office, 
Philadelphia, 

Pennsylvania 

Auditor General, De- 
fense Supply Agency: 

Physical Inven- 11/24/72 
tot: y Procedures 
and Practices 

Medical Supply 9/ 5/73 
Functions 


Mobilization 3/18/73 

Reserve Re- 
quirements '<t 
Defense Supply 
Centers 

Veterans Administra- 
tion, Fiscal Audit: 

Audit of On-Job 5/ 8/74 

and Apptentice- 
sh ip Training 
Awards Processed 
by OCR 

Processing Oe- 9/17/73 

pendency Changes 
from Supplemental 
Award Code Sheets 


APPENDIX V 

Type of . 

application Problem identified 

involved Software Data' 


Requisition X 

processing 
and redistr i- 
bution 

Overhaul sched- X 

uling and re- 
distribution 


Physical inven- 
tory requests X 


Customer re- X 

turns, requisi- 
tion processing 
and stock attri- 
tion 

Customer returns.' X 


Payments X 


Payments ' X 
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Title of 

reoort Date 


Type of 

application Problem iden tif ied 

involved Software “Data 


Veterans Adraini stra- 
ti on, ' Fiscal Audit 
{ continued } r 

Processing Awards 6/ 2/73 Payments X 

after Entitlement 
is Exhausted 

Nonrecovery of 4/20/73 Payments X 

Accounts Receiv- 
able from Re- 
sumed BCL Ac- 
count Payments 

Retroactive 8/31/73 Payments X 

Payment Adjust- 
ments 

Updating Ac- 4/12/73 Payments X 

counts Receiv- 
able Deduction 
Amount from 
Amended Awards 


Duplicate Chap- 8/ 2/74 Payments X 

ter 34 Educa- 
tion Payments 


Interior, Office of 
Survey and Review, 

Audit Operations: 

Review of Con- 10/29/73 Payments X 

tract No. 

N00C1420S253 
Kith the Navajo 
Tribe, Window 
Rock, Arizona, 

Bureau of In- 
dian Affairs 

Agriculture, Office 1 

of Inspector General: 

Programs Option 10/25/73 Payments X 

B Provisions of 

the 1972 Feed 

Grain Set-aside 

Program 
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Title of 

• repot t Date 

Agriculture, Office 
of Inspector General 
(continued ) : 

Loading Order 8/ 8/73 

Issuance Proc- 
essing and Set- 
tlement 

Agriculture, Office ' 

of Aud it: 

Automated Ac- 2/15/74 

counting Service 

HEW Audit Agency: 

Administrative 1/ 9/74 

Costs Incur: ed. 
and Benefit Pay- 
ments Made Under 
the Health In- 
surance for the 
Aged Act 


Administrative 12/28/73 
Costs Incurred 
and Benefit Pay- 
ments Made Under 
the Health In- 
surance for the 
Aged Prograr 

Administrative ; 6/23/74 

Costs Proposed 

and Operations 

Relating to Ben- 

ef i t Payments 

Under Medicare 

Administrative 5/24/74 

Costs Claimed 

and Benefit 

Payments Made 

Under the 

Health insurance 

for the Aged 

Program 


APPENDIX V 

Type of 

application Problem iden tified 

involved Software Data 


Loading order 

settlement X X 


Payments and X X 

bi 1 1 i ngs 

Payments. X 


Payments. X 


Payments X 


Payments X 
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‘Title of 
report 

am -Audit Agency, 
(continued) : 

Administrative 
Costs claimed 
and Supplementary 
Medical Insurance 
. Benefit Payments- 
M a de U nd e i H e a 1 1 h 
Insurance for the 
Aged Program 

Admin i st rat i ve 
Costs and Bene-' 
fit Payments 
Under .the Health 
Insurance for 
the Aged Program 

Administr at i ve* 
Costs Claimed 
and Benefit 
Payments Made ' 
Under t h e Hoa 1th 
Insurance tor 
the* Aged Pi on ram . 

Admin J st i at i ve ■ 
Costs and Benefit 
Payrsen t s Made Un- 
der the Health 
Insurance for 
the Aged Act 


Date 


* . 


74 


Type of 
application 
i nvol ved 


APPENDIX V 


Problem identified 


r . of tware 


Dat 


11/ 9/73 Payments 


11/12/73 Payments 


Payment s 


Payments 
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